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ABSTRACT 


The U.S. military is the largest single user of simulation in the world, and our visual simu¬ 
lations can be software-intensive systems with a lifespan of many years. Managers of these 
simulations need tools to help them make better decisions at the architectural level. Cur¬ 
rently, no such quantitative models with supporting metrics exist for this purpose. There are 
properties that are held as positive characteristics in visual simulation architectures. Visual 
simulation architectures can be distinguished from one another based on three characteris¬ 
tics: (1) openness, as defined by the use of standards, licensing, and support of innovation; 
(2) reuse, as defined by the potential of being used in subsequent projects; and (3) agility, 
as defined by the ease with which software can be integrated, reconfigured, or repurposed. 
In this research, we propose quantifiable models to measure openness, reuse, and agility, 
and claim that the models adequately distinguish visual simulation frameworks from one 
another. Furthermore, we claim that these models can enhance military acquisition deci¬ 
sions. The results show that application of the metrics offers a level of granularity that is 
useful in identifying key differences in simulation frameworks that could have profound 
downstream implications. 
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I. INTRODUCTION 


This research is about choosing a visual simulation architecture in acquisition. “Ar¬ 
chitecture” is referring to such properties as object oriented, service oriented, component 
based, data driven, blackboard, and peer-to-peer. Acquisition is a big problem in the De¬ 
partment of Defense and is always being improved. In acquisition, simulation is sometimes 
used to help evaluate competing products. Visual simulations are a narrower field within 
that. Acquisition professionals have simulations to help them choose products, but they do 
not have quantitative tools to help them choose visual simulation architectures. 

A. THESIS AND PROBLEM STATEMENTS 

Visual simulation architectures can be distinguished one from another based on 
three objectives: (1) openness, as defined by code’s use of standards, its licensing, and sup¬ 
port of innovation; (2) reuse, as defined by code’s ease of being used in subsequent projects; 
and (3) agility, as defined by the ease with which code can be integrated, reconfigured, or 
repurposed. 

In our research, we found no existing quantitative models with metrics to help assess 
these objectives in visual simulation architectures. We developed three quantifiable models 
to measure openness, reuse, and agility and claim that the models adequately distinguish 
visual simulation frameworks from one another. Furthermore, we claim that these metrics 
are sufficient for improving military acquisition decisions. 

B. BACKGROUND 

The United States (U.S.) military is the largest single user of simulation in the world 
(McDowell, Darken, Sullivan, & Johnson, 2006). Military visual simulations are software¬ 
intensive systems that may have a long lifespan and address diverse military situations. 
In acquisition, a simulation sometimes supports a program, and sometimes the simulation 
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is the program. Military acquisition involves a wide range of products, services, tactics, 
doctrines, customers, vendors, timelines, and budgets that must be considered. Whether 
simulation is the central or supporting role, costs associated with simulation can be enor¬ 
mous. Therefore, having methods to help make good decisions are of critical importance. 

Cost and safety concerns related to product development drive acquisition organi¬ 
zations to choose simulation, e.g., synthetic environments, as a way to experiment with 
new ideas, mitigate risk, or objectively compare alternatives. When the human element is a 
critical part of the acquisition process, assessment becomes even more complex due to hu¬ 
man capital availability, hardware and software changes, and the training needed to ensure 
success. Modem simulation tools and prototyping utilities are often expensive, inflexible, 
and proprietary. 

In developing a new ship or airplane, a Program Manager (PM) may have the budget 
to create a massive simulation from scratch, but most in the acquisition community have to 
budget for simulation carefully. They have many programs to manage, many stakeholders 
to consider, and many pockets of experience scattered throughout the community and even 
within the program office. Their best hope is that they can build a simulation quickly and 
cheaply with good justification. 

Even for well-funded acquisition programs, efficiency and cost effectiveness are 
important. Except for the case where the simulation itself is the program, every dollar spent 
on Modeling and Simulation (M&S) is a dollar not spent on other forms of engineering and 
product development. Defense M&S often has difficulty making a business case for itself 
because of this. 

Program managers need simulation software that enables them to address a variety 
of needs, incorporate expertise from different sources, build trust in the simulations among 
users, answer questions quickly, and save money. It is rare for a program to start a major 
simulation project from scratch. Programmers use frameworks. Application Programming 
Interfaces (APIs), toolkits, libraries, etc., as a starting point for fast and cheap develop¬ 
ment. The understanding in the Department of Defense (DoD) about the importance of 
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architectures has improved, but there were no quantifiable tools to assist PMs in selecting a 
framework that represents a desirable architecture. The research presented in this disserta¬ 
tion did not reveal any previously-published scientific methodology or model using metrics 
for assessing simulation software. The requirement is legitimate and unfilled. 

There are a variety of qualities in visual simulation software that a PM may desire. 
The three qualities of openness, reuse, and agility chosen for study in this dissertation are 
not comprehensive, but they are representative and supported in the literature. If code 
survives a long time, maintainability may also be important. Perhaps certain technical 
features are desired. Security is a concern—or should be—for many systems. Although 
a PM may desire all of these, some are more important to a project than others. Visual 
simulation fosters the sharing of data, whether models of equipment or terrain databases, 
so openness, reuse, and agility are of particular concern. 

The specific needs for individual PMs and programs differ. Consider the following 
three scenarios with three PMs with different programs and different simulation needs (Ta¬ 
ble 1). One PM has a simulation for a major combat system. This is a long-term view. The 
system is expected to be around for many years, and participants will change over time. 
Another PM is in a different situation. Here there is a game based maintenance trainer for 
the F-22. If successful, the general is going to say, “That’s great! Let’s put it out for the 
Joint Strike Fighter too. Maybe we can get some of their funding.” Finally, there is the PM 
with a UAV exercise coming up. There is a new camera to put on. Mission planners wonder 
how the flight profiles might change with a different sensor. It is no surprise that there is 
no one architecture or product that would be best for all of these situations, but what do the 
PMs have to help them choose an architecture? The three objectives of openness, reuse, 
and agility are good things on which to inform a decision. 

The problem addressed in this dissertation is the absence of a quantifiable model, a 
tool, to aid PMs in selecting an appropriate visual simulation framework. The models de¬ 
veloped here do not address all of the concerns a PM may have. Many cannot be addressed 
in a single dissertation. However, the models do provide a rigorous and quantifiable as- 
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Table 1: Characteristics of the three example programs drive different needs in the visual 
simulation software used in the programs. 


Major Combat System 

Game Based 
Maintenance Trainer 

UAV Sensor Exercise 

15 year timeline 

Forks expected 

Quick & dirty 

Vendor longevity 

Licensing 

Focused question 

Technology changes 

Configuration 

Pre-packaged 

Maintenance costs 

Manpower 



sessment of three important elements of visual simulation software: openness, reuse, and 
agility. 


C. ORGANIZATION OF THIS DOCUMENT 

Chapter II reviews the literature to show that there are no existing models available 
for assessing the important issues of openness, reuse, and agility for visual simulation. 
Regarding openness, the literature reveals a mixed set of issues. A taxonomy presented 
by Maxwell (2006) provides a good basis for our model. Openness is divided into three 
main attributes: standards, licensing, and innovation. A model presented by Anvaari and 
Jansen (2010) provides structure for our model. Regarding reuse, the literature reveals 
some established software metrics from S. Chidamber and Kemerer (1991). These metrics 
are respected and are connected to many positive software attributes including ease of reuse. 
Regarding agility, the literature reveals a less cohesive picture and no metrics on which to 
base a model. 

Chapter III presents our openness model, how it was developed, with a study veri¬ 
fying the ability to distinguish between two visual simulation frameworks. Using the tax¬ 
onomy of issues presented by Maxwell (2006) and the layers and operations presented by 
Anvaari and Jansen (2010), this research develops criteria for assessing openness on a three- 
axis system of layers, operations, and issues. The categorical ratings for each point can be 
weighted according to a user value system to provide numerical assessment, if needed. The 
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model is applied to two visual simulation frameworks representing two very different ar¬ 
chitectures. The model successfully differentiates the frameworks. It draws attention to 
important distinguishing attributes. 

Chapter IV presents the reuse model, how it was developed, and a study verifying 
its ability to differentiate two visual simulation frameworks. Using the Chidamber and 
Kemerer (CK) metrics and empirical validation studies found in literature, this research 
develops criteria for the metrics. The categorical ratings for each metric can be weighted 
according to a user value system to provide numerical assessment, if needed. The model is 
applied to two visual simulation frameworks representing two very different architectures. 
The model successfully differentiates the frameworks. It is weaker than the openness and 
agility models. 

Chapter V presents the agility model, how it was developed, and a study verify¬ 
ing its ability to differentiate twoscriptsize visual simulation frameworks. Unlike the reuse 
metrics, which can be calculated on static source code, agility metrics must be measured 
“in motion.” There must be an act by a developer, and the effort must be estimated. This 
research develops metrics for estimating the effort of swapping out a portion of code and 
applies these metrics to two visual simulation frameworks representing two very different 
architectures. The agility model reveals a dramatic difference between the two architec¬ 
tures. 

Chapter VI is the conclusion and presents a discussion about the research, a sum¬ 
mary of findings, and considerations for future work. 


5 



THIS PAGE INTENTIONALLY LEET BLANK 


6 



II. LITERATURE REVIEW 


This chapter presents our literature review and identifies openness, reuse, and agility 
as important features in software development for visual simulation. Common through all 
three attributes is a general agreement that they are desirable but that there exists a lack of 
detail about how to measure them. 

Openness is defined by the use of standards, licensing, and support of innovation, 
is considered an important quality in visual simulation software. It is discussed in many 
papers inside and outside the visual simulation realm. Many people have opinions, asser¬ 
tions, and definitions for openness, but no models suitable for measuring the openness of 
visual simulation architectures could be found. 

Reuse is defined by the ease of code being used in subsequent projects. Therefore, 
reuse is considered an important quality in visual simulation software. Effective code reuse 
has been a goal of computer programmers since subroutines were invented in 1949 for the 
EDS AC computer (Wilkes & Renwick, 1949). While we found many individual metrics 
for different aspects of reuse, we found no models of reuse that could encompass these 
metrics to supply a decision maker with differentiating results. 

Agility is defined here as the ease with which code can be reconfigured, repurposed, 
or integrated. It is also an important goal in visual simulation software. As military threats 
change, software requirements also change (Eanman & Proctor, 2009; Scott, 2010), and 
therefore, simulation software must be easily repurposed for unanticipated use. Agility is 
the least rigorously defined and studied of the three features assessed in this dissertation. 
This literature was the most difficult for two reasons: (1) agility means different things to 
different people, and (2) our meaning of agility is sometimes only identifiable in an author’s 
work by discerning the author’s intent and examining his or her actual actions. No models 
suitable for measuring the agility of visual simulation architectures could be found. 
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A. OPENNESS 


1. What is Openness? 

There are many definitions and attributes of openness provided in the literature. 
A review of them reveals many common traits. While reviewing these many facets of 
openness, we use three overarching issues—standards, licensing, and innovation—that will 
support our openness model. This taxonomy of openness is used by Maxwell (2006) in 
his defense of the idea that value can be created through openness. All of the literature 
reviewed below focuses on one or more of these aspects of openness. 

Gimenes, Silva, Reis, and Oliveira (2008) describe flight simulation environments 
for unmanned aerial vehicles. They address standards, licensing, and innovation with their 
definition of openness. Their definition includes defined APIs, communication protocols, 
data formats, viewing and altering source code, and community expansion through mod¬ 
ules. 

The Open Systems Joint Task Force (Open Systems Joint Task Force, 2004) ad¬ 
dresses openness by providing guidance for program managers to implement a Modular 
Open Systems Approach (MOSA), which they call the “fundamental building block of 
joint integrated warfare systems.” The Defense Acquisition Guidebook gives five MOSA 
principles, which address standards, licensing, and innovation: (1) establishing an enabling 
environment, (2) employ modular design, (3) design key interfaces, (4) use open standards, 
and (5) certify conformance (Defense Acquisition University, 2010). 

a. Standards 

Some literature highlights the role of standards with respect to openness. 
Here openness is defined by its relation to such issues as accessibility, integration, commu¬ 
nication, and interoperability. 

Describing the phases for developing a domain-oriented reuse library for 
the U.S. Air Force, Curfman (1993) addresses standards by identifying accessibility as 
an attribute of openness, which he connects to policy and communications. In the same 
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year Oswalt (1993), in his detailed report on “Current Applications, Trends, and Organiza¬ 
tions in U.S. Military Simulation and Gaming,” defines standardization as the “number of 
standards to which a particular simulation conforms” and recommends the military adopt 
commercial standards. 

Writing for the Command, Control, Communications, Computers, Intelli¬ 
gence, Surveillance and Reconnaissance (C4ISR) Cooperative Research Program, Krygiel 
(1999) addresses standards by studying integration issues on large scale systems. She con¬ 
nects open systems with interfaces, data formats, hardware portability, and interoperability. 

The Open Systems Joint Task Force (2004) says that standards must be 
“well defined, mature, widely used, and readily available” and that standards should be 
selected based on “maturity, market acceptance, and allowance for future technology in¬ 
sertion.” Their preferences for the adoption of standards is (1) open standards, (2) de facto 
standards, and (3) government and proprietary standards. 

Maxwell (2006) identifies standards with communication carrier and con¬ 
tent. This he applies both to networking, with Transmission Control Protocol/Internet 
Protocol (TCP/IP) being an example of carrier and Hypertext Markup Language (HTML) 
being an example of content. He also calls attention to the ability of software to run on 
disparate hardware, as is the case with American National Standards Institute (ANSI) C or 
the Portable Operating System Interface for Unix (POSIX) standard. 

b. Licensing 

Open licensing refers to what a consumer is permitted to do with a piece 
of software. Licenses can be very restrictive, such as requiring a student-licensed copy of 
software not to be used for commercial purposes, or very open, such as the entire source 
for a system being made available for review and modification. 

The struggle is between the creators, who have the right to control their 
work, and the users, who may see themselves as prevented from doing things with software 
that they might wish to do (Maxwell, 2006). The government’s understanding of this bal- 
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ance is further evidenced in its opposition to certain kinds of monopolies but endorsing of 
limited-term, “mini-monopolies” with the patent and copyright systems. 


The DoD Open Source Software (OSS) Frequently Asked Questions web¬ 
site (Department of Defense, 2011) identifies open source as “software for which the 
human-readable source code is available for use, study, re-use, modification, enhancement, 
and re-distribution by the users of that software.” 

Muller (2011) divides software licensing into seven categories: (1) Public 
domain, (2) Free Software, (3) Open Source Software, (4) Freeware, (5) Shareware, (6) Pro¬ 
prietary Software, and (7) Patented Software. Each of these has certain characteristics that 
our model should differentiate. 

Different organizations differ on their distinction between free and open 
source. Richard Stallman, an advocate for free software, encourages people to think in 
terms of “free speech” rather than “free beer.” His meaning of free software refers more 
to the rights granted to users of software (Stallman, 2010). The Open Source Initiative’s 
definition of open source differs on some points from Stallman’s definition of free (Open 
Source Initiative OSI, 2011). Asa result,/ree is sometimes distinguished from open source 
(as with Muller’s taxonomy above). Sometimes it is grouped together, as with the common 
acronym Free and Open Source Software (FOSS), which acknowledges that a distinction 
exists but that there is more in common than not. 

The DoD is also careful to distinguish among different meanings of free and 
open. Chief Information Officer for the Department of Defense David M. Wennergren, in 
a memorandum dated 16 October 2009 with the subject “Clarifying Guidance Regarding 
Open Source Software (OSS),” clarifies that the DoD Instruction 8500.2, which limits the 
use of software with “limited or no warranty,” does not apply to open source software, for 
which anyone could provide maintenance and a warranty because the source code is avail¬ 
able. David Wheeler, who helped draft this memorandum, pointed out in an interview that 
freeware is no-cost, while closed-source is software for which there is no one to provide 
maintenance or a warranty. Therefore, open source software cannot be considered free- 
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ware (Schwartz & Phipps, 2011). The memorandum goes on to explain that “In almost all 
cases, OSS meets the definition of ‘commercial computer software’” and provides six le¬ 
gal references. Because contractors are generally required to prefer “commercial computer 
software” over writing their own software from scratch, this clarification gives contractors 
greater latitude in selecting open source software in their work. 

Maxwell (2006) recalls that in the early days of computing in the 1950s and 
1960s, software was written primarily in academic settings where sharing ideas and infor¬ 
mation was expected but that it was a community norm, not a legal requirement. As people 
began to think in terms of owning software, open source licenses, which grant explicit per¬ 
mission to view and modify source code, arose. Open source licenses focus on the rights 
of the users and widest possible dissemination. 

c. Innovation 

Open innovation refers to a community being able to collaborate and share 
ideas. The moniker “innovation” is not always used in the literature. The innovation that 
people desire from openness is revealed in words like participation, sharing, and collabo¬ 
ration. 

In an article entitled “The Many Faces of ‘Open,’” Updegrove (2005) relates 
attributes of openness that he claims are “generally conceded.” Three features of openness 
to which he calls attention are participation (who is involved), process (participants abil¬ 
ity to influence the outcome), and terms (intellectual property rights). In this Updegrove 
acknowledges both the innovation and the licensing aspects of openness. 

For Maxwell (2006), the defining characteristics of open innovation are col¬ 
laboration and sharing. He warns that this should not be confused with a lack of intellectual 
property or the abolishment of compensation but shows that new forms of compensation 
emerge when open innovation takes hold. Open innovation is a community being able to 
collaborate and share ideas and software. 
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2. Why Openness is Important 

Openness is important to software development and visual simulation software in 
particular. A review of the literature reveals many advantages of openness. A common 
thread is that big ideas from individuals or small teams can find their way to a wide audience 
when openness is encouraged. Innovators are able to leverage the work done by others, 
standing on the shoulders of giants. 

a. Benefits 

Openness is of immense importance in the world of the acquisition profes¬ 
sionals who seek to find the best solutions for the government. They may need to merge 
solutions from many sources and may find themselves having to integrate partial solutions 
to achieve their goals. In the case of simulation, with the right software framework, they 
may be able to forge their own simulations, leveraging the work of others and contributing 
back the portions unique to their domain. 

Describing the process of modifying Commercial Off-the-Shelf (COTS) 
games for military use, Fong (2004) notes the shift from paying game developers huge 
sums of money and submitting to stringent licensing agreements to modifying existing 
games with mods. He cites many benefits that this example of open innovation brings. For 
the game developer, third party mods extend the shelf life of their games bringing economic 
benefit. Those writing mods benefit from a low barrier to entry. With low risk and low cost, 
they get to leverage sophisticated game technology. Mods offer quicker turnaround time 
in bringing new capabilities to bear than creating a new game from scratch. Experienced 
soldiers, marines, sailors, and airmen in the community of gamers can identify inaccuracies 
and report them resulting in higher fidelity models. Fong identifies the biggest obstruction 
is a lack of source code and hopes that more developers will embrace the open source move¬ 
ment: “If more game developers adopt the same mantra of free information access, it will 
pave the way for more extensive modifications of COTS games to meet specific military 
interests.” 
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The Open Systems Joint Task Force (2004) lists many benefits of openness 
for program managers. They encourage program mangers to embrace MOSA because it is 
an enabler to help them make affordable changes, employ spiral development, develop in¬ 
tegrated roadmaps for their projects, adapt to evolving threats, leverage commercial invest¬ 
ment, reduce development time, improve interoperability, mitigate risk from obsolescence, 
and mitigate risk of a single supply source, to name a few. 

Wichmann (2002) reports on the state of open source software in businesses 
and points out that openness in the standards creation process is critical to reduce the pos¬ 
sibility that a standard is used to keep certain players out of a market. Open standards 
increases competition and helps ensure a level playing field for all participants. 

Open standards encourage participation by many parties and reduce the like¬ 
lihood that only a single party’s interests are represented. One company denying access to 
a specification to a competitor may be attractive to the company in power, but it is a dis¬ 
advantage to consumers who may resist being locked into one provider because of a lack 
of open standards. Greater participation and greater adoption spurs greater innovation of 
benefit to consumers (Maxwell, 2006). 

The Economist (2005) noted that open standards “allow and promote unex¬ 
pected forms of innovation.” They cite several examples where people have made mash-ups 
that create value by joining information from one website with information from another. 
This also serves to boost the popularity of the source websites. The economic benefit pro¬ 
vided by mash-ups is necessary because, “if the information being mashed is useful, it is 
probably expensive for the originated sites to put on the web in the first place.” 

WaughPartners and OSSWatch (2007) show that openness impacts sustain¬ 
ability, applicability, interoperability, and trustworthiness. Sustainability is improved for 
data by open data standards and in software by open source licensing. This helps pro¬ 
tect against data loss and software obsolescence. Applicability (who is able to benefit 
from software) is improved by replacing a single point of control—and their single set of 
requirements—with open access and collaboration allowing more people to add value to 
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the system. Interoperability is improved with open communication standards. TCP/IP is 
a prime example. Trustworthiness (knowing that a system will perform as documented) is 
improved by open source licensing where interested parties can perform their own audits 
on software. 

The Open Technology Development Roadmap Plan (OTDRP), prepared for 
the Deputy Under Secretary of Defense, identifies open standards and open interfaces as 
key elements in the success of DoD software (Herz, Lucas, & Scott, 2006). It states that 
the “DoD must pursue an active strategy of open interfaces, modularity, and reuse” and 
outlines a strategy to combine “salient advances” in the following areas, which are directly 
addressed by this research (Figure 1). 


Open Technology Development combines salient 
advances in the following areas: 

1. Open Standards and Interfaces 

2. Open Source Software and Designs 

3. Collaborative/Distributive culture and the 
and online support tools 

4. Technological Agility 


Figure 1: The advances recommended by the Open Technology Development Roadmap 
plan (Herz et ah, 2006) are supported directly by this research. 


The OTDRP also identifies advantages that openness (and reuse and agility) 
in software development contribute to national security: 

• Enhances agility of Information Technology (IT) industries to more rapidly adapt 
and change to user needed capabilities. 

• Strengthens the industrial base by not protecting industry from competition. Makes 
industry more likely to compete on ideas and execution versus product lock-in. 

• Adoption recognizes a change in our position with regard to balance of trade of IT. 
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• Enables DoD to secure the infrastructure and increase security by understanding what 
is actually in the source code of software installed in DoD networks. 

• Rapidly respond to adversary actions as well as rapid changes in the technology 
industrial base. 

Several authors write that openness fosters interoperability, spurs competi¬ 
tion that benefits consumers, increases participation, increases opportunities for success, 
and avoids vendor lock in (Davis & Anderson, 2004; Maxwell, 2006; McDowell et ah, 
2006; Scott, 2010). Maxwell (2006) acknowledges that some people argue that open stan¬ 
dards may reduce efficiencies that tightly integrated proprietary systems can offer and that 
open standards may reduce innovation because developers are no longer forced to think 
outside the box as they are forced to work around proprietary technology. This is a poor ar¬ 
gument, and he asserts that standards permit innovation to be focused where the real value 
lies, in integrating systems. 

Regarding innovation. Democratizing Innovation (Hippel, 2005) and The 
Wealth of Networks: How Social Production Transforms Markets and Freedom (Benkler, 
2006) explore the value added to individuals and a society by collaboration and sharing. 
They show that win-win scenarios are possible when innovative ideas are shared. In this 
way innovation mirrors money in an old proverb that might be updated to read, “Innovation 
is like manure. Unless you spread it around, it doesn’t do much good.” 

Open innovation also blurs the line between producer and consumer. Con¬ 
sumers know their needs but cannot always articulate them well. However, give someone 
capable tools, and they may solve their own problem. The online auction site eBay is an 
example of a company that provides the means for users to solve their own problems—and 
thousands of people now make their living with their own “eBay Store.” 
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b. Mandates 


Not to be overlooked are government mandates related to openness. Be 
assured that these mandates are in place because of the benefits provided by openness, but 
as mandates or government “suggestions” they receive special recognition here. 

The Clinger-Cohen Act of 1996 directed the U. S. Government to operate 
more business-like with respect to information systems. (Clinger & Cohen, 1996) Accord¬ 
ingly Department of Defense Directive (DoDD) 5000.01 requires the use of modular, open 
systems. Enclosure 1.1.27 states that, “Acquisition programs shall be managed through 
the application of a systems engineering approach that optimizes total system performance 
and minimizes total ownership costs. A modular, open-systems approach shall be em¬ 
ployed, where feasible” (emphasis added) (Department of Defense, 2003). DoD Instruc¬ 
tion 5000.02 follows up in Enclosure 12.8: “MODUEAR OPEN SYSTEMS APPROACH 
(MOSA). Program managers shall employ MOSA to design for affordable change, enable 
evolutionary acquisition, and rapidly field affordable systems that are interoperable in the 
joint battle space” (Department of Defense, 2008). 

Individual services have followed up with their own guidance directing the 
use of modular, open systems. Expanding on DoDD 5000.01, then United States Assistant 
Secretary of the Navy John J. Young, Jr., in a letter dated 5 August 2004 with the subject 
“Naval Open Architecture Scope and Responsibilities,” wrote that he was initiating “an ef¬ 
fort to establish open architecture principles as the basis for all war fighting systems devel¬ 
opment and maintenance” (Young Jr., 2004). The Deputy Chief of Naval Operations Rear 
Admiral M. J. Edwards, in a letter dated 23 December 2005 with the subject “Requirement 
for Open Architecture (OA) Implementation,” established “the requirement to implement 
Open Architecture (OA) principles across the Navy Enterprise” and cited as one justifi¬ 
cation the need to “implement agile changes that support rapidly evolving requirements” 
(Edwards, 2005). 

The “Clarifying Guidance Regarding Open Source Software (OSS)” mem¬ 
orandum referred to earlier identifies open source software as an enabler to “anticipate new 
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threats and respond continuously to changing environments” (Wennergren, 2009). This 
memorandum identified many advantages of open source software such as increased secu¬ 
rity through peer review, faster response times through unrestricted ability to modify source 
code, reduced barriers to entry and exit for vendor participation, and cost savings by reduc¬ 
ing per-seat licensing and a shared maintenance burden. This memorandum clarifies many 
misunderstandings in the DoD about open source software. 

DoD Instruction 5000.2-R Mandatory Procedures for Major Acquisition 
Defense Programs and Major Automated Information Systems Acquisition Programs iden¬ 
tifies openness in system design as crucial to enable full and open competition (Office of 
the Under Secretary of Defense, 2002). Regarding standards, it states that “M&S standards 
facilitate reuse... and reduce cost by providing approved solutions to common problems.” 

The OSD Acquisition Modeling and Simulation Master Plan (AMSMP) di¬ 
rects the development of open standards in systems engineering and architecture modeling 
as well as standards for distributed, simulated environments (Office of the Under Secretary 
of Defense, 2006). 

In Modeling and Simulation in Manufacturing and Defense Systems Ac¬ 
quisition the National Research Council recommends a culture of collaboration, part of 
openness, in DoD acquisition (National Research Council, 2005). 

3. How Openness is Measured 

There has been some effort to quantify openness at different levels. The literature 
has both one-off suggestions for openness metrics as well as in-depth surveys to assess 
licensing details. This literature review did not reveal any openness models that would 
meet our requirements for differentiating visual simulation architectures. 

Regarding standards, the Open Systems Joint Task Force (2004) encourages pro¬ 
gram managers to use specific program measurements to gauge a program’s progress in 
implementing MOSA. It even goes so far as to suggest a metric for openness: “For exam¬ 
ple, the percentage of key interfaces defined by open standards could be used as a metric to 
measure the degree of system openness.” 
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WaughPartners and OSSWatch (2007) present their “first draft” of an openness 
evaluation model and provide a survey with 46 questions regarding licensing, standards, 
knowledge, governance, and marketplace along with numeric values for the responses to 
the questions in order to calculate an openness value for each of the five categories. They 
apply their model to three open source Unix-like kernels and three databases (two open 
source and one proprietary). The issues addressed in this work fit roughly within our tax¬ 
onomy of openness—standards, licensing, and innovation—but while it addresses openness 
in great detail, it addresses software projects as a whole and does not distinguish between 
the different parts of a system and degrees of openness within each part, which we show in 
this dissertation is a strong differentiator in visual simulation. 

Anvaari and Jansen (2010) evaluate five mobile phone operating systems and intro¬ 
duce evaluating openness at a finer grained level. They evaluate the software along two 
axes, which they call layers and factors. Layers refers to the function and scope of the soft¬ 
ware pieces. Factors refers to possible development operations performed on the software. 
Their evaluation consists of determining whether each factor is possible to perform at each 
layer and the associated licensing implications. Standards and innovation are not part of 
their evaluation. 

Focusing on licensing, Muller (2011) evaluates 20 Integrated Library Systems (ILS) 
for the purpose of finding the best open source programs for library use. He throws out all 
systems that he does not qualify as open source. To determine what is open, he measures 
the “correlation between the practices within the community and the terms associated with 
free or open software license” and divides software licenses into seven categories from 
public domain to patented. Differentiation by license is his first concern. He continues his 
ILS evaluation by considering the communities behind each project and almost 800 specific 
functions and features related to library use. 

4. Summary of Openness 

Openness has many facets, but the three major headings of standards, licensing, and 
innovation capture most people’s concerns and represent the openness we wish to measure 
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in visual simulation architectures. In all the literature reviewed, no models suitable for 
measuring the openness of visual simulation architectures could be found, though some 
features on which a model could be built are present. 

B. REUSE 

I. What is Reuse 

There are varying forms of reuse and meanings used in the literature, and since 
some confusion may arise when definitions are too narrow, we present some explanation 
of the term reuse. The simplest forms of reuse have been around the longest and have the 
most widespread use. More sophisticated forms of reuse arose later and sometimes have 
limited applicability or are tied to specific architectures. Ultimately, the literature focuses 
on reuse as a measure of code quality. Many metrics have been presented to this end, and 
most assume Object Oriented (00) programming. 

Ambler (1998) breaks out reuse into eight different types: (1) Code, (2) Inheri¬ 
tance, (3) Template, (4) Component, (5) Framework, (6) Artifact, (7) Pattern, and (8) Do¬ 
main Component reuse. Making use of the visual simulation frameworks discussed in this 
dissertation would involve code, inheritance, and framework reuse. 

Reuse through patterns is considered a high form of reuse because it refers not 
to actually reusing code but reusing ideas or approaches to solving problems. In this 
sense patterns permit reuse to happen even across programming languages. In the land¬ 
mark book Design Patterns: Elements of Reusable Object-Oriented Software the so-called 
“gang of four” identify such common patterns as Singleton, Adapter, Iterator, and Observer 
(Gamma, Helm, Johnson, & Vlissides, 1995). 

J. Lewis (2006) identifies several examples of coarse reuse in military simulations. 
These include purchasing off the shelf games, e.g., Microsoft Flight Simulator and Falcon 
4.0, modifying existing games, e.g.. Spearhead II and Marine Doom, and paying to have 
game companies develop custom simulations with their underlying engines, e.g., America’s 
Army. 
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Mcllroy (1968), at a now-famous North Atlantic Treaty Organization (NATO) con¬ 
ference in Garmisch, Germany, proposed a market for “mass produced software compo¬ 
nents” where pieces of code could be bought and sold. He recognized that no one com¬ 
pany or even industry would be able to produce a full complement of quality software but 
that each could contribute its best work. Although Mcllroy’s specific vision has not ma¬ 
terialized, parts of his ideas can be seen in the emergence of both a multi-billion dollar 
commercial software industry as well as the open source software community. 

Jansen, Brinkkemper, Hunink, and Demir (2008) define two kinds of reuse that de¬ 
scribe the developer more than the code. Pragmatic reuse is the extension of third party 
software that was not found with any formal procurement method and may not have been 
designed with reuse in mind. Opportunistic reuse is extending software with third party 
software that was not meant to be integrated or reused. Though others prefer more delib¬ 
erate and systematic reuse (Morad & Kufiik, 2005; Ommering, 2005), Jansen et al. show 
how two companies benefit from even this kind of ad hoc reuse. 

2. Why Reuse is Important 

Reuse is considered by many to be a key to improving software productivity and 
quality (Biggerstaff & Richter, 1987; Kim & Stohr, 1992; Mill, Mill, & Mill, 1995), and 
reuse is mandated or strongly encouraged in the DoD. However, after many years of re¬ 
search and advances in techniques and metrics, reuse is still not as common as many people 
would like (Biggerstaff & Richter, 1987; Herz et al., 2006; Garlan, Allen, & Ockerbloom, 
2009; Scott, 2010). 


a. Benefits 

For most people it is intuitive that code reuse is a good thing, and when 
pressed, two main reasons are likely to emerge: time and money (Washizaki, Yamamoto, 
& Fukazawa, 2003; Haefliger, Krogh, & Spaeth, 2008). Of course, the two are related, and 
a saving of either one is of great interest (Becker, 1965). Development savings can range 
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from 50-100% with typical savings being about 80%, which can pay off even when it costs 
extra to write the initial code with reusability in mind (Poulin, 2006). 


Reuse can reduce the effort required of developers (Mill et ah, 1995; Davis 
& Anderson, 2004; Ragab & Ammar, 2010) by amplifying the software developer’s ca¬ 
pabilities and reducing the number of symbols in a system (Biggerstaff & Richter, 1987) 
resulting in higher quality software (Mili et ah, 1995). Brutzman, Zyda, Pullen, and Morse 
(2002) write, “Interoperable reuse is essential for feasibility, life-cycle supportability, fund- 
ability, and product flexibility.” 

Analyzing his company Philips, which manufacturers software-intensive 
hardware such as medical systems and consumer electronics, Ommering (2005) cites three 
main challenges, which he shows are improved by good reuse: (1) increasing complexity in 
the software, (2) growing diversity in products, and (3) a decrease in allowed development 
time. These challenges should be familiar to any program manager in the Department of 
Defense. He finds that their systematic and component based approach to reuse results in a 
more manageable software base than the “opportunistic” reuse that marked the early days 
of electronics manufacturing. The component subsystems have a longer lifespan than the 
actual products they support, and Philips has developed a set of golden and silver rules with 
varying degrees of effectiveness at reducing unintended consequences. Ommering and oth¬ 
ers (Biggerstaff & Richter, 1987) observe the balance in writing software general enough 
to be reused but specific enough to be useful. He believes that their deliberate efforts at 
systematic reuse have significantly benefitted the company. 

Software is often not reused in the DoD (Herz et ah, 2006; Scott, 2010). 
This results in wasted funding with multiple development efforts. This also results in slower 
development times making it harder for the DoD to respond to new missions and emerging 
threats. 


b. Mandates 

The government has opinions regarding reuse as well, and military services 
have instituted programs to promote and manage code reuse. Early efforts include the Air 
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Force’s Central Archive for Reusable Defense Software (CARDS), the Navy’s Restructured 
Naval Tactical Data System (RNTDS), the Army’s Reusable Ada Product for Information 
System Development (RAPID) program, and the Defense Software Reuse System (DSRS) 
(Therriault & Van Nederveen, 1994). 

Air Force Instruction (AFI) 33-114 states, “Software reuse benefits the Air 
Force through increased developer productivity, improved quality and reliability of software¬ 
intensive systems, enhanced system interoperability, lowered program technical risk, and 
shortened software development and maintenance time” (U.S. Air Force, 2004). 

In “Open Technology Development (ODT): Lessons Learned and Best Prac¬ 
tices for Military Software” Scott (2011) emphasizes the need for code reuse as a necessary 
means for developing software quickly and inexpensively. The OTDRP also recommends 
reuse. The AMSMP assumes the need for reuse and lays out requirements for systems 
enabling the discovery of reusable software and data (Office of the Under Secretary of 
Defense, 2006). 

3. How Reuse is Measured 

We can divide the measurement of reuse into two categories: actual reuse and po¬ 
tential for reuse. Actual reuse refers to completed projects that have reused code from 
previous projects, such as the OTDRP suggesting DoD program managers count the num¬ 
ber of times a software component has been used by multiple (acquisition) programs (Herz 
et ah, 2006). Potential for reuse, with which this dissertation is more concerned, refers to 
the ease with which code might be reused, based on various qualities of that code. 

Briand, Daly, and Wust (1999) defined two kinds of attributes that help describe 
software: internal and external. Internal attributes can be defined in terms of the software 
itself (e.g., lines of code). External attributes cannot be measured solely in terms of the 
software (e.g., comprehensibility). Some external factors that affect reusability include 
comprehensibility and maintainability (Cho, Kim, & Kim, 2007). 
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Complexity, coupling, and cohesion are three internal attributes that support reusabil¬ 
ity and are related to comprehensibility, maintainability, quality, and the separation of func¬ 
tion and purpose (Table 2). 

Selecting among 00 metrics is a weighty task. Xenos, Stavrinoudis, Zikouli, and 
Christodoulakis (2000) list over 89 metrics that can and have been applied to 00 software 
and more metrics have been published since then (Table 3). Despite the many metrics put 
forth, only a few have lasted beyond a few papers. 

S. Chidamber and Kemerer (1991) proposed metrics to address object oriented soft¬ 
ware specifically. Their work was immediately taken up by others. Li and Henry (1993a) 
refine some ambiguities in the CK metrics and propose two more detailed metrics to replace 
the CK metric regarding class coupling. 

Again regarding coupling, Martin (1994) distinguishes between individual classes 
and categories of classes and introduces Instability, a ratio describing how many classes 
inside a category depend on classes outside the category. 

The stability of the software involved in the coupling also affects the degree to 
which coupling may have a practical effect (White, 1994; Hitz & Montazeri, 1995). Cou¬ 
pling to basic language implementations such as integers are less troublesome than cou¬ 
pling to small foundation classes such as string or date, and all of these couplings are 
preferred over coupling to classes in the problem domain. 

The most enduring software metrics related to reusability and other qualities come 
from S. R. Chidamber and Kemerer (1991, 1994). Their six “CK” metrics have been often 
put to the test and are still used today (Cho et ah, 2007). They address concerns that too 
many software metrics, especially for object oriented design, do not have solid theoretical 
basis (Kearney, Sedlmeyer, Thompson, Gray, & Adler, 1986), lack theoretical rigor (Vessey 
& Weber, 1984), refer to procedures rather than objects (Henry & Kafura, 1984), and do not 
possess appropriate mathematical properties for producing “normal predictable behavior” 
(Prather, 1984; Weyuker, 1988). The CK metrics are Weighted Methods per Class (WMC), 
Depth of Inheritance Tree (DIT), Number of Children (NOC), Coupling Between Object 
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Table 2: Researches agree that complexity, coupling, and cohesion affect code comprehen¬ 
sibility, maintainability, quality, and therefore reusability. 


Paper 

Quote 

Mills, 1988 

The results indicated that module coupling was an important factor in determining the quality 
of the resulting product. 

Wand & Weber, 1990 

It is generally believed that system decompositions which have “loosely-coupled” subsystems 
are easier to understand than system decompositions which have “tightly-coupled” subsystems. 

Devanbu, Brachman, & 
Selfridge, 1991 

Thus, this lack of knowledge among developers leads to a vicious cycle where the system be¬ 
comes progressively more complex, and thus harder to know. 

S. Chidamber & Kemerer, 
1991 

Excessive coupling between objects outside of the inheritance hierarchy is detrimental to mod¬ 
ular design and prevents reuse. 

Sharble & Cohen, 1993 

The recognized achievement of OOSD is the production of software that is less complex, and is 
therefore easier to maintain and extend, and can be more easily reused. 

Sharble & Cohen, 1993 

Excessive coupling between objects outside of the inheritance hierarchy is detrimental to mod¬ 
ular design and prevents reuse. 

Hitz & Montazeri, 1995 

Software engineering experts assure that designs with low coupling and high cohesion lead to 
products that are both, more reliable and more maintainable. 

Briand, Morasca, & Basili, 
1996 

Lower complexity is believed to provide advantages such as lower maintenance time and cost. 

Briand, Daly, Porter, & 
Wust, 1998 

Some measures, in particular coupling and inheritance ones, are shown to be significantly re¬ 
lated to the probability of detecting a fault in a class during testing. 

Allen & Khoshgoftaar, 

1999 

When used in conjunction with measures of other attributes, coupling and cohesion can con¬ 
tribute to an assessment or prediction of software quality. 

Tang, Kao, & Chen, 1999 

Excessive coupling indicates weakness of module encapsulation and may inhibit reuse. 

Agrawal, Bayardo, Gruhl, 

& Papadimitriou, 2002 

Such loose-coupling of distributed components reduces coordination overhead, fostering faster 
parallel development. 

Open Systems Joint Task 
Force, 2004 

Decoupling modules eases development risks and makes future modifications easier. 

Xu, Qian, & He, 2006 

The decoupling metrics can be used to measure and evaluate the decoupling attributes of a 
distributed, service- oriented software architecture that has very significant impacts on the un- 
derstandability, maintainability, reliability, testability, and reusability of software components. 

Cho et al., 2007 

Excessive coupling is detrimental to modular design and prevents reuse. The more independent 
a class is, the easier it is reuse in another application. 

Offutt, Abdurazik, & 
Schach, 2008 

Software coupling can be used to estimate a number of quality factors, including maintainabil¬ 
ity, complexity, and reliability. 
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Table 3: A sampling of the over eighty metrics which have been proposed to measure 
software (Xenos, 2000). 


Metric Description 

AHF Attribute Hiding Factor is the ratio of the sum of inherited attributes in 
all system classes under consideration to the total number of available 
classes attributes. 

CEC Class Entropy Complexity measures the complexity of classes based on 
their information content 

CLM Comment Lines per Method measures the percentage of comments in 
methods. 

DAM Data Access Metric is the ratio of the number of private attributes to the 
total number of attributes declared in the class. 

FOC Function Oriented Code measures the percentage of non object-oriented 
code that is used in a program. 

INP Internal Privacy refers to the use of accessory functions even within a 
class. 

MAA Measure of Attribute Abstraction is the ratio of the number of attributes 
inherited by a class to the total number of attributes in the class. 

MHF Method Hiding Factor is defined as the ratio of the sum of the invisibil¬ 
ities of all methods defined in all classes to the total number of methods 
defined in the system under consideration. 

NAD Number of Abstract Data types is the number of user-defined objects used 

as attributes in a class that are necessary to instantiate an object instance 
of the class. 

NCM Number of Class Methods in a class measures the measures available in 

a class but not in its instances. 

NPM Number of Parameters per Method is the average number of parameters 

per method in a class. 

PCM Percentage of Commented Methods is the percentage of commented 
methods. 
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Classes (CBO), Response for a Class (RFC), Laek of Cohesion in Methods (LCOM). These 
metries will be examined in greater detail in the Chapter IV: Reuse. 

Rosenberg and Hyatt (1995) summarize several papers and books that propose met¬ 
ries to measure various software qualities ineluding “understandability, reusability, and 
maintainability,” all three of whieh are linked in their estimation. They identify supporting 
papers that eonfirm the relationship between the metries and the various software qualities. 
Of the metries that pertain to understandability, reusability, and maintainability, they eite 
Size with four supporting works. Comment Pereentage with one supporting work, WMC 
with four supporting works, RFC with four supporting works, LCOM with five supporting 
works, CBO with six supporting works, DIT with five supporting works, and NOC with 
four supporting works. 

Kitehenham (2010) surveys the literature again and finds that CK metries dominate 
the researeh. Although she and a few others take issue with some of them, these metries 
are the most studied and most understood metries that we ean “reuse” to help us measure 
reuse. 

The CK metries have been validated by a number of studies (Li & Henry, 1993b; 
Basili, Briand, & Melo, 1996; S. Chidamber, Darey, & Kemerer, 1998; Tang et ah, 1999; 
Cartwright & Shepperd, 2000; Olague, Etzkom, Gholston, & Quattlebaum, 2007). Most 
of these studies eorrelated the CK metries to the fault-proneness of the eode, a response 
variable that eould be measured by examining the ehange history of a eodebase. These 
validation studies also provide an empirieal basis for estimating expeeted values of the 
metries in software. 

4. Summary of Reuse 

Reuse has been estimated by many metries by many people, but the most enduring 
are the CK metries. The literature overwhelmingly speaks of metries for reuse in terms 
of eode quality. This is not the meaning of reuse that we originally intended, and this 
foreed the introduetion of agility as a third model. In all the literature reviewed, no models 
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suitable for measuring the reuse of visual simulation architectures could be found, though 
some metrics on which a model could be built are present. 

C. AGILITY 

1. What is Agility 

We refer to agility specifically as software being easily reconfigured, repurposed, or 
integrated, but there are many other definitions to sift through in the literature. Qumer and 
Henderson-Sellers (2006b) writes, “there is no rigorous or complete definition of agility.” 
Dove (1994) writes, “agility is a very seductive word” and describes a litany of “personal 
definitions” that may accompany it: cycle time reduction, customization, streamlining, 
reengineering, learning organization, productivity. Scott (2010) uses the term adaptability. 
Tech Web (2008) uses the expression, “react quickly to changing market dynamics.” Several 
authors, especially in the Journal of Defense Modeling and Simulation, instead use the term 
“composability” to refer to agility (Yilmaz, 2004; Davis & Anderson, 2004). The Defense 
Acquisition Guidebook links agility to integration and optimization (Defense Acquisition 
University, 2010). 

For the Office of Naval Research (ONR), Tangney (2009) desires to make “inter¬ 
esting perturbations” of Naval tasks under which they would like to make measurements in 
a calibrated Synthetic Environment for Assessment (SEA). This reconfiguring of an SEA 
represents agility. 

Agile Methods is different from agility. Agile Methods is a movement in software 
development that favors collaboration, interaction, and responsiveness over “documenta¬ 
tion driven, heavyweight software development processes” (Beck, Cockburn, Jeffries, & 
Highsmith, 2001; Cohen, Eindvall, & Costa, 2004). Agile Methods may also have value in 
visual simulation development, but that is not the focus of this dissertation. 

2. Why Agility is Important 

Scott (2010) defends the accusations that the U.S. government lacks imagination but 
instead contends that “we are simply unable to deploy new ideas as effectively or as quickly 
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as we could.” He cites industrial examples of large-scale agility while the DoD spends tens 
of billions of dollars annually that is “rarely reused and difficult to adapt to new threats.” 
Lynn III (2010) writes, “Cyberwarfare is like maneuver warfare, in that speed and agility 
matter most.” 


a. Benefits 

Davis and Anderson (2004) give several reasons why agility in software is 
important to defense M&S. They provide their reasons as assertions, acknowledging that a 
body of literature and the common sense of practitioners agree. Software modules that are 
easily reconfigured, repurposed, or integrated are also easier to build at the creation phase. 
Such software tends to be easier to understand. It makes testing easier when modules can 
be pulled out and tested on their own. Cost can be reduced. 

McDowell et al. (2006) point out that agility also helps mitigate risk when 
technologies mature at different rates. They also highlight the need to adapt simulations to 
different genres of simulations and different real world domains from air to land to sea. 

In studying Service Oriented Architectures (SOA) in businesses, several au¬ 
thors highlight the positive impact of agility (TechWeb, 2008; Feig, 2008). Agility is linked 
to efficiency in software development and faster time to market. 

The OTDRP (Herz et ah, 2006) highlights the need to adapt to new “trends, 
capabilities, and practices.” By falling behind in software, the DoD has seen costs spiraling 
up and a loss of useful software to those on the ground. It encourages the DoD to create 
market incentives to increase agility, but they offer no metrics with which to achieve this. 

b. Mandates 

As mentioned in Section b, the Deputy Chief of Naval Operations saw a 
need to “implement agile changes that support rapidly evolving requirements” and to that 
end wrote a policy letter directing open architecture principles across the Navy (Edwards, 
2005). 
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The AMSMP links a loss of agility in software to a lack of ability to develop 
live-virtual-constructive environments that exploit the full range of hardware, software, 
ranges, equipment, and other resources that are available. The result is simulations that are 
less capable than they could be (Office of the Under Secretary of Defense, 2006). 

3. How Agility is Measured 

We found very little literature regarding any measurements of agility. The metrics 
that we did find are tangentially related and help provide context, but our literature review 
did not reveal any agility models that would meet our requirements for differentiating visual 
simulation architectures. 

Qumer and Henderson-Sellers (2006b) measures agility in organizations using Ag¬ 
ile Methods. He develops four dimensions to measure. One dimension that is closest to our 
needs is agility characterization with metrics for Flexibility, Speed, Leanness, Learning, 
and Responsiveness. These metrics have a value of 0 or 1 for various phases and segments 
depending on the answer to a yes/no question provided for each metric. For example, the 
question associated with Flexibility is, “Does the method accommodate expected or unex¬ 
pected changes?” During a planning phase, flexibility may be a 1 but leanness may still be 
0. Adding up these values, and dividing by the total number of measurements taken, gives 
a fractional value indicating degree of agility. Qumer and Henderson-Sellers (2006a) use 
this methodology to analyze two Agile Methods known as XP and Scrum for the purpose 
of helping organizations select among competing Agile Method approaches. 

4. Summary of Agility 

Agility means different things to different people. Therefore, a review of the lit¬ 
erature required casting a much broader net over related topics. Still in all the literature 
reviewed, no models suitable for measuring the agility of visual simulation architectures 
could be found. 
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D. SUMMARY OF LITERATURE REVIEW 


This chapter reviewed literature that identifies openness, reuse, and agility—important 
features in software development, especially as it relates to visual simulation. There is some 
agreement in definitions and importance, and there were related efforts at measuring these 
factors, but we found no quantitative models suitable for our use in measuring openness, 
reuse, and agility in visual simulation architectures. 

What the literature did reveal are some foundations on which we can build assess¬ 
ment models with confidence. These features of openness, reuse, and agility are not new, 
and there is great interest in them. These foundations, built over many years by many peo¬ 
ple, enable assessment models to be built for this dissertation. The literature provided least 
benefit with respect to assessing the agility of visual simulation frameworks. 
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III. OPENNESS 


In this chapter, we develop and apply a model for assessing openness based on the 
definition and need established in Chapter II: Literature Review. The model borrows from 
taxonomies and methodologies presented in literature and presents a new composite model 
that is subsequently shown to differentiate between two visual simulation frameworks. 

A. DEVELOPING THE OPENNESS MODEL 

To assess visual simulation frameworks, we want a model that can differentiate 
between the parts within a framework, the actions we might take, and do all this across 
various issues associated with openness. We assess on a three axis system with four layers 
of software, three operations of development, and three issues in openness. 

1. Layers and Operations 

Anvaari and Jansen (2010) develop a model for assessing the openness of mobile 
phone operating systems. For each of four layers of the architecture they consider three 
factors, whether or not the factors are possible, and the licensing restrictions associated 
with them. 

Their breakdown of layers and factors can be applied to visual simulation software 
despite the fact that it was developed for mobile phone software. The four layers they 
identify are Kernel, Middleware, Native Applications, and Extended Applications. The 
three factors they identify are integrating (use existing components), extending (enhance 
functionality of components), and modifying (replace a component) the platform (Figure 
2 ). 

We use the four layers given by Anvaari and Jansen (2010) and define the layers in 
the context of visual simulation frameworks: 

• External Applications. The final simulations and applications built with the frame¬ 
work. 
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Figure 2: A model that considers the effort and permissions required to integrate, extend, 
or modify a platform (From Anvaari & Jansen, 2010). 


• Internal Applications. Applications or tools included with the standard distribution 
used either in the building or running of the simulation. 

• Middleware. The standard pieces that are included with the framework, often called 
“foundation classes” in the case of a programming language. 

• Kernel. The core engine that manages the simulation pieces. 

The model does not include a holistic analysis of openness, but it measures whether 
or not an operation can be performed at a given layer and if so, the licensing restrictions 
therein. We retain the operational definitions of Integrate, Extend, and Modify: 

• Integrate. To use the existing pieces of a layer via API, service call, source code 
inclusion, shared data object, and other software calling mechanisms. An example 
would be using the cURL libraries to make HTTP requests. 

• Extend. To enhance the functionality of the pieces of a layer beyond just making use 
of that layer. An example would be to change the behavior of a camera-following- 
an-object routine. 

• Modify. To replace or change the pieces of a layer. An example would be replacing 
the physics engine. 


32 

























Anvaari and Jansen finish their model with a summary of the “possibility” mea¬ 
surements they made (Figure 3). Their summary provides both a quiek look assessment for 
differenees among the platforms as well as details that ean be inspeeted at partieular points 
along the axes. 
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P = Possibility, L = Licensing Status, Po = Possible, Pc = Possible for 
some components, Np = Not possible, Pn = Pennission is not needed, Ps 
= in some cases permission is needed. Pa = Permission is always needed 


Figure 3: Anvaari & Jansen (2010) analyze mobile smart phone arehiteetures with their 
openness model, whieh we use as a starting point for our model whieh is more all- 
eneompassing of the notion of openness. 


They find that not all organizations are eoneerned about all the layers and opera¬ 
tions. For many developers, as long as they ean integrate, extend, or modify their own 
extended applieations, that is suffieient for them. This helps to explain why developers 
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continue to create iPhone (now iOS) apps despite the cries of open source advocates that 
the developers’ hands are shackled on that platform. One developer humorously explains 
that “if a platform is open enough to make a lot of money for him, then the platform is 
interesting for him.” 

2. Issues 

Maxwell (2006) defends the idea that value can be created through openness and 
presents a useful taxonomy of openness: open standards, open licensing, and open innova¬ 
tion. These three issues within openness form the basis for our third axis for analysis. 

Open standards refer to the communication mode and content of a system. This 
might include a program’s application programming interface (API), the data formatting 
and syntax offered in the Extensible 3D (X3D) graphics format, or the binary byte ordering 
of the Internet Protocol (IP). There is such a need for standards that there are many national 
and international standards bodies addressing various technical fields such as the Internet 
Engineering Task Eorce (lETE) for the Internet and the International Telecommunication 
Union (ITU) for telephone communication. 

Open licensing refers to what a user is permitted to do with a piece of software. 
Eicenses can be very restrictive, such as requiring a student-licensed copy of software not 
be used for commercial purposes, or very open, such as the entire source for a system being 
made available for review and modification. 

Open innovation refers to a community being able to collaborate and share ideas 
and software. Big ideas from small parties can find their way to a wide audience when 
innovation is encouraged, and innovators are able to leverage the work done by others, 
standing on the shoulders of giants. 

• Standards. The communication mode and content of a system. Examples would be 
a program’s API, an XME schema, or TCP/IP. 

• Licensing. What a user is permitted to do with a piece of software. A user may be 
restricted in use or even permitted full access to source code. 
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• Innovation. A community being able to collaborate and share ideas and software. 
This spans both the propensity for the software to encourage collaboration as well as 
actual collaboration happening within the using community. 

3. Criteria 

For each layer and each operation, criteria are established for assessing the open¬ 
ness of each operation on each layer with respect to standards, licensing, and community 
innovation. Unlike the Anvaari and Jansen model, which asks if it is possible to integrate, 
extend, or modify each layer, we ask how does the product’s handling of standards, licens¬ 
ing, and innovation help to integrate, extend, or modify each layer. 

This model uses categorical classifications to assess openness but avoids the value 
judgements embedded in the red, yellow, green color scheme in favor of green, yellow, 
blue. 


a. Standards 

In assessing how standards affect an operation on a layer, we must consider 
the kind of software and methods of integration that are offered, if any. In a software 
library, open might mean having integration take place by documented API calls that were 
developed in collaboration with many parties. A less open alternative might be documented 
API calls that are closed to changes aside from the framework’s vendor. Not open at all 
might mean not being able to integrate or only through unpublished or private APIs. In 
contrast a web oriented framework might regard open as having integration take place over 
Hypertext Transfer Protocol (HTTP) with Extensible Markup Language (XML) data in a 
Simple Object Access Protocol (SOAP) message whose formatting is discovered in the 
Universal Description, Discovery, and Integration (UDDI) and which was published by the 
World Wide Web Consortium (W3C). 

With respect to standards, the following classifications are used: 

• Is = The techniques for operating on a layer are based on documented, open stan¬ 
dards that include community participation. 
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• IIs = The techniques for operating on a layer are partially standardized, or the stan¬ 
dard is not subject to community participation. 

• Ills = There are no techniques for operating on a layer or the techniques involve 
unsupported “hacks” or direct source code editing. 

b. Licensing 

In assessing how licensing applies to an operation on a layer, we consider 
permissions both for redistribution and access to different parts of the framework. The gov¬ 
ernment has an interest in making sure it has access to all parts of a framework necessary 
and that models developed by one organization are not locked away from other govern¬ 
ment organizations—or even different projects in the same organization—due to licensing. 
We might expect to see less differentiation among the layers and operations here than in 
standards since software tends to be licensed en masse, not piece by piece. 

Closed source software often has no provision for viewing its source code. 
Some vendors may charge a fee and require a non-disclosure agreement before allowing 
access to source code. Some vendors, whether open or closed source, make a good deal of 
sample code available to help developers at the application layer. Sometimes it is possible 
to integrate into the kernel of a framework through documented APIs (some open standards, 
IIs) but have no access to the kernel source code. 

With respect to licensing, the following classifications are used: 

• It = Users have the right to access, modify, and redistribute both their finished 
simulation and the framework’s source code without express permission. 

• lit = Users are restricted in how they may access, modify, or redistribute either their 
finished simulation or the framework and its source code. 

• lilt = Users may not redistribute their finished simulations or have no access to the 
framework’s source code. 
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c. Innovation 

In assessing how innovation applies to an operation on a layer, we eonsider 
both the eneouragement offered to eollaborate as well as the eollaboration that is already 
happening. The government is interested in the benefits that many smaller aequisition 
programs ean eontribute to everyone, and with a thriving eeosystem with eollaboration and 
shared ideas and produets, the entire eommunity is enriehed. 

Assessing a framework based on its eommunity involvement may seem un¬ 
fair to the framework, but the goal is not just to assess a pieee of software on its own but 
rather how appropriate the framework is for use in aequisition simulation. This holistie 
approaeh eenters on the needs of the user rather than on trying to isolate the framework 
from its intended audienee. This also means that “involvement” must be defined within the 
proper seope. Niehe markets would not be expeeted to have the same number of innovating 
partieipants as something with mass-market appeal. New frameworks with few users will 
need to be assessed based on the innovation demonstrated within the eurrent user base. 

With respeet to innovation, the following elassifieations are used: 

• It = Software and vendor lends itself to and a eommunity takes advantage of inno¬ 
vation. 

• III = Software or vendor may diseourage or a eommunity may not be greatly inter¬ 
ested in innovation. 

• nil = Software or vendor restriets or there is no eommunity of interest seeking 
innovation. 

d. Summary of Criteria 

The openness eriteria is summarized in Table 4. For eaeh issue, the eriteria 
are applied to eaeh layer and operation. 
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Table 4: Assessment criteria for the three openness three issues that are applied to the four 
layers and three operations. 


Issue 

Rating 

Criteria 


Is 

The techniques for operating on a layer are based on documented, 
open standards that include community participation. 

Standards 

IIs 

The techniques for operating on a layer are partially standardized, 
or the standard is not subject to community participation. 


Ills 

There are no techniques for operating on a layer or the techniques 
involve unsupported “hacks” or direct source code editing. 


II 

Users have the right to access, modify, and redistribute both their 
finished simulation and the framework’s source code without ex¬ 
press permission. 

Licensing 

III 

Users are restricted in how they may access, modify, or redis¬ 
tribute either their finished simulation or the framework and its 
source code. 


nil 

Users may not redistribute their finished simulations or have no 
access to the framework’s source code. 


li 

Software and vendor lends itself to and a community takes advan¬ 
tage of innovation. 

Innovation 

Hi 

Software or vendor may discourage or a community may not be 
greatly interested in innovation. 


nil 

Software or vendor restricts or there is no community of interest 
seeking innovation. 
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4. Model for Assessing Openness 

Building on the model developed by Anvaari and Jansen (2010), we present a visual 
model that assesses openness both at a glanee, in color and weight of markers, and in detail 
(Table 5). This follows the principles of small multiples, which encourages a visual cue to 
be repeated along axes and present both detailed information and information at a glance 
(Tufte, 2001). 

The decision maker gains insight both in the process of applying the model and in 
examining the model. In applying the model, a deep and structured understanding of the 
models is developed. In examining the model, a decision maker may study the various as¬ 
pects of proposed frameworks and consider where a less-open framework may be tolerated 
without adding unnecessary risk. 

Table 5: A model for assessing the openness of simulation frameworks. 


Framework 1 Framework 2 


Layer 

Operation 

Std 

Eic 

Inn 

Std Eic Inn 

External 

Integrate 

Is 

Ii 

Ii 


Extend 

ITc 

III 

nil 

Hi 


Applications 

Modify 

Ills 

iiii 



Internal 

Applications 


Integrate 

Extend 

Modify 


Integrate 

Middleware Extend 
Modify 


Integrate 

Kernel Extend 

Modify 

Std, s = Standards; Lie, I = Licensing; Inn, i = Innovation; 
I, II, III are classifications. 
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5. Weights for User Assigned Value Systems 

If a final numeric score is desired, weights can be assigned to the categories accord¬ 
ing to what layers, operations, or issues are most important. When summed, these weights 
provide for a tailored assessment of the frameworks. Here is where value can be applied, 
because the user specifies what is important. 

Let L be the set of layers L = {External Applications, Internal Applications, Mid¬ 
dleware, Kernel} and I G L be a layer. Let O be the set of operations O = (Integrate, 
Extend, Modify} and o G O be an operation. Let I be the set of issues (Standards, Li¬ 
censing, Innovation} and i G I be an issue. Let Rtoi be the categorical rating assigned 
to a framework at the given layer, operation, and issue. Let be the weighting 

function that returns the user assigned value for a given Rioi- Then the overall openness 
value Vo of a framework is given by Equation 1: 

Vo = LZL ^lot(^loi) (1) 

leL oeo iei 

The development of these weights for a particular use case is beyond the scope of 
this dissertation, but they may be used to help score frameworks against the specific needs 
of a program manager. 

B. STUDY 1: ASSESSING OPENNESS 

To demonstrate the feasibility of this assessment model, two simulation frame¬ 
works DeltaSD and DMZ were selected and assessed using the methodology presented 
here. Of the hundreds of game engines available (M. Lewis & Jacobson, 2002), these two 
frameworks were selected because they represent two fundamentally different approaches 
to building visual simulation frameworks, often called game engines. They are also both 
readily available for download and inspection. The study demonstrates the feasibility of 
measuring openness to distinguish visual simulation architectures. 
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1. Methodology 

After selecting the frameworks, the layers were identified and the criteria applied. 
Applying the model to the frameworks required an understanding of the frameworks in¬ 
volved. A careful study of the two frameworks preceded this study. 

The layers of the software were defined for each framework. Specific tools, names¬ 
paces, folders, source code, and other pieces of the frameworks were identified. Clarity 
was required to ensure that the criteria were applied cleanly to the layers without one area 
bleeding over into another. 

With the criteria laid out, each combination of layer, issue, and operation was ex¬ 
amined. The criteria determined the categorical ratings to assign. 

The ratings were compiled into a table, which is presented as a whole as well as 
divided into areas of interest, according to the distinctions made by the model. 

2. Delta3D 

DeltaSD (Figure 4) is a successful open source game engine developed at the Naval 
Postgraduate School (NPS) in Monterey, CA. It has over six years and $1 million of de¬ 
velopment behind it. Its staff of programmers both maintain the framework and use the 
framework for research projects. It is also used by other organizations around the world. It 
boasts many features and integrates a number of open source libraries such as Open Scene 
Graph for rendering. Open Dynamics Engine for physics, and OpenAL for audio, to name 
a few. 

The DeltaSD source consists of 160,000 lines of code and nearly 1,200 classes. It 
is written in C-i-i-. Table 6 lists the major namespaces into which DeltaSD is divided and 
the size of each. The programming team uses good object-oriented programming prac¬ 
tices. The code is representative of the conventional approach to game engine develop¬ 
ment. Much of the development team had prior experience with the Unreal engine by Epic 
Games, one of the largest game engines on the market. Consequently, DeltaSD was de¬ 
veloped following many Unreal paradigms. Classes are structured reasonably around the 
problem domain with the ubiquitous “Actor” class tying much of the functionality together. 
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Figure 4: DeltaSD is an open source game engine used by many projects around the world 
and has a staff of programmers. 


Table 6: DeltaSD consists of many classes divided into namespaces related to their purpose. 


Namespace 

Classes 

Lines of Code 

dtABC 

31 

3,381 

dtActors 

98 

10,616 

dtAI 

97 

8,101 

dtAnim 

55 

6,535 

dtAudio 

14 

2,927 

dtCore 

155 

24,204 

dtDAL 

111 

13,397 

dtDirector 

98 

15,080 

dtDIS 

16 

1,589 

dtGame 

103 

9,555 

dtGUI 

19 

3,206 

dtHLAGM 

46 

9,128 

dtInspectorQt 

21 

2,243 

dtLMS 

13 

711 

dtNet 

3 

360 

dtNetGM 

11 

1,971 

dtQt 

46 

6,409 

dtScript 

1 

57 

dtTerrain 

54 

5,055 

dtUtil 

96 

10,577 

NA 

57 

1555 

psGeodeTransform 

1 

18 

sigslot 

43 

2090 
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DeltaSD also comes with a number of “helper programs,” whieh we eall Internal 
Applieations (Figure 5). One is ealled STAGE (Figure 5a), whieh helps in developing envi¬ 
ronments with buildings, terrain, aetors, and more and ean be thought of as a type of “level 
editor” for DeltaSD. The objeets deseribed in STAGE ean be manipulated programmati- 
eally by ealling the various aetors’ funetions. A new reeently-released tool ealled Direetor 
(Figure 5b) provides a graphieal environment for even non-programmers to seript behaviors 
in a simulation. 



(a) DeltaSD’s STAGE (b) DeltaSD’s Director 

Figure 5: DeltaSD has powerful tools to help build simulations sueh as STAGE for building 
environments and Direetor for seripting behaviors. 


3. DMZ 

DMZ (not an aeronym) (Eigure 6) is a new open souree, eomponent-based game 
engine developed at Naval Postgraduate Sehool (NPS). It grew from a frustration that 
so many student projeets eould not be easily reintegrated into one souree tree beeause of 
the fragility of the eode. The elasses and funetionality developed by students touehed too 
many other parts of the software eausing a dependeney quagmire. DMZ developer Randall 
Barker ereated a new game engine foeused on developing small, reusable ehunks of eode 
that eenter around funetionality and behavior rather than objeet eneapsulation. 

What he “invented” was serviee-oriented, eomponent-based programming applied 
to visual simulation, though he explains it as just a natural engineering solution to the 
elassie dependeney problem he had always fought in GO programming. Note how well 
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this maps to the recommendation made by the Open Technology Development Roadmap 
Plan (Herz et ah, 2006): “This report recommends shifts in the process of technology 
acquisition from closed, locked-in black box systems to open and modular approaches. 
These approaches are based on open standards, services based architectures, open source 
collaboration, and reference open source implementations.” 



Figure 6: DMZ is a new open source, component-based game engine used at NPS. 


The DMZ source consists of over 98,000 lines of code and over 400 classes in over 
800 files. It is written in C-i-i-, but simulations can also be constructed in the JavaScript or 
Lua scripting languages. Although the architecture is component-oriented, it is still built 
with classes and objects. All DMZ code resides in a single namespace. The developers 
instead group code in directories. Table 7 lists the major headings into which DMZ is 
divided and the size of each. 

It is interesting and perhaps not surprising to observe that there are a greater number 
of verb class names than normally found in object oriented programming (Table 8). In 
DeltaSD 5.5% of the classnames end in verbs, while in DMZ 9.7% of the classnames end 
in verbs. While not a scientific conclusion, this does suggest that DMZ is service (verb) 
oriented instead of object (noun) oriented. 

DMZ does not have much in the way of Internal Applications, as it is not as mature 
as DeltaSD and has a smaller staff of developers. There are some scripts that generate 
boilerplate code. These scripts probably contribute to some of the higher-than-expected 
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Table 7: DMZ consists of many classes divided by directory according to their purpose. 


Directory 

Classes 

Lines of Code 

frameworks/archive 

11 

2,530 

frameworks/audio 

11 

1,962 

frameworks/entity 

28 

6,849 

frameworks/event 

10 

2,317 

frameworks/input 

18 

2,659 

frameworks/net 

27 

8,754 

frameworks/obj ect 

27 

5,727 

frameworks/qt 

73 

14,185 

frameworks/render 

45 

10,238 

frameworks/weapon 

6 

1,288 

foundation/libs 

26 

1,879 

foundation/plugins 

10 

1,760 

kemel/runtime 

68 

5,427 

kemel/system 

19 

1,484 

kemel/types 

31 

2,459 


Table 8: The object-oriented DeltaSD tends to be organized around nouns and adjectives, 
while the service-oriented DMZ tends to be organized around verbs. 


Framework 

Sample Class Names 


BaseWaterActor 

Delta3D 

AiAcioxRegistry 

Animatable 

SonnACommand 

SkyDome 

DMZ 

EntityPluginArricM/ate 

EntityPluginFo//ow 

InputPluginChannelSw/tc/z 

Oh]eci¥\ugmHighlight 

ObjectPlugin5e/ect 


45 



Weighted Methods per Class metrics since the scripts generate a number of placeholder 
methods that are often never used. Plans to create a “level editor” type of tool exist, but 
no such tool was available during this research for assessment. There is a “make” system 
called Imk for compiling components. The system is written in a scripting language called 
Lua (Imk = Lua make). This is used extensively in the development of DMZ simulations. 

4. Identifying Software Layers 

A clear delineation must be made for each of the four layers. Some layers will have 
a good deal of code behind them while others may be minimal or non-existent. Table 9 
maps the four layers to specific parts of the frameworks. 

Table 9: Layers for the two frameworks DeltaSD and DMZ are broken out to aid in assess¬ 
ing openness. 


Framework 

Layer 

Description 


External 

The final simulations and applications built with the 


Applications 

framework. 

DeltaSD 

Internal 

Applications 

Tools stored in utilities folder (AlUtility, Anima- 
tionViewer, Exporters, GameStart, EMS, MapDump, 
ObjectViewer, ParticleEditor, STAGE). 


Middleware 

Code not in the Kernel namespaces. 


Kernel 

Code in the dtCore, dtGame, and dtDAE namespaces. 


External 

The final simulations and applications built with the 


Applications 

framework. 

DMZ 

Internal 

Scripts in the scripts folder and the Eua “make” 

Applications 

(imk) system. 


Middleware 

Code in the frameworks folder. 


Kernel 

Code in the foundation and kernel folders. 
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5. Applying Criteria 

The process of applying the criteria requires stepping through each combination of 
layer and operation to compare the three issues against the criteria established earlier. 

a. External Applications 

In both frameworks the openness of the the external applications is deter¬ 
mined in large part by the developer of the final simulations, not the framework itself. 
Whether Extending, Integrating, or Modifying, a common theme is that there is little influ¬ 
ence by the framework on what developers do. 

Delta3D 

With respect to Standards, developers Integrating, Extending, or Modifying 
another developer’s External Applications are not guaranteed to have access via open stan¬ 
dards. DeltaSD neither requires nor forbids that developers use open standards, and there 
is no mechanism in the DeltaSD architecture itself to enable it. Therefore, only a rating of 
IIs is appropriate for DeltaSD for Integrating, Extending, or Modifying. 

With respect to Eicensing, developers working with DeltaSD are bound by 
the Eesser GNU Public Eicense (EGPE). This license does not require that developers 
using DeltaSD to produce External Applications release these applications as open source. 
However, they are not forbidden from releasing these applications as open source either. 
DeltaSD External Applications are rated IIi, for licensing for Integrating, Extending, or 
Modifying. 

With respect to Innovation, there is no provision in DeltaSD to encourage 
developers of External Applications to seek collaboration and sharing within their appli¬ 
cations. Although it is not prohibited, no evidence could be found of it occurring. Appli¬ 
cations built with DeltaSD are likely to be “one way” applications that are built once and 
not used by anyone else. DeltaSD External Applications are rated Illi for innovation for 
Integrating, Extending, or Modifying. 
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DMZ 


With respect to Standards, developers Integrating, Extending, or Modifying 
External Applications have an advantage in DMZ. They can tap into the Object module, 
which manages the data defining the world of the simulation, or the Event module, which 
manages communication among modules and plugins. Even without published information 
for an External Application, the very architecture of DMZ enables developers to Integrate, 
Extend, or Modify applications in the same way regardless of the source. The standardized 
XME configuration files and the inspectable Object and Event modules aid developers in 
interacting with applications in a standard and straightforward way. Integrating, Extending, 
and Modifying DMZ External Applications are rated Is for Standards. 

With respect to Eicensing, developers working with DMZ are bound by the 
Massachusetts Institute of Technology (MIT) Eicense, one of the shortest of all open source 
licenses. It levies few requirements on developers except the need to give credit to the 
original DMZ developers and to hold DMZ blameless. Again, developers are not required 
to or forbidden from releasing External Applications as open source. Therefore, DMZ 
External Applications are rated lli, for licensing for Integrating, Extending, or Modifying. 

With respect to Innovation, the very architecture of DMZ makes innovation 
of External Applications possible. Applications are made up of composable elements that 
can be shared and on which developers can collaborate. However, DMZ being a new de¬ 
velopment and mostly used in-house, the only evidence of innovation is in a very small 
community. DMZ External Applications are therefore rated Hi for innovation for Inte¬ 
grating, Extending, or Modifying. 

b. Internal Applications 

The characteristics of Internal Applications are directly influenced by the 
frameworks themselves. The nature and quality of the Internal Applications affect how 
developers are able to make use of the frameworks, and their openness can have lasting 
effects. 
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Delta3D 


With respect to Standards, DeltaSD has numerous tools to aid in the devel¬ 
opment of rich 3D worlds. They consist mostly of standalone tools, but they interoperate 
through shared data files. The STAGE tool for example supports the . ive file type, which 
comes from the open source Open Scene Graph (OSG) rendering engine. Unfortunately 
the . ive file type does not have a published specification, is defined only in the imple¬ 
menting code within OSG, and is now obsolete (OSGForum, 2011). The replacement file 
type . osg also has no published file format. Despite this, the community still uses . osg 
files as one of many standard Three Dimensional (3D) file formats. 

Integrating with Delta3D Internal Applications consists primarily of loading 
data files, and as such Delta3D makes a good effort to support a number of standard, either 
open or ^(e/acto, file types and is rated Is for Standards. 

Extending or Modifying Delta3D Internal Applications is a different matter 
altogether. There is no approved technique for extending their behavior. The code is open 
source, fortunately, but that is a different issue. There is no plugin architecture or scripting 
or any technique (other than manipulating source code) for making these changes to Inter¬ 
nal Applications. Extending or Modifying Delta3D Internal Applications are rated lllj 
for Standards. 

With respect to Eicensing, the Delta3D Internal Applications are released 
under the EGPE (or the GNU Public Eicense (GPE) in the case of STAGE which uses the 
Qt library, also released under GPE). As such developers are not restricted in their use 
or redistribution. Integrating, Extending, or Modifying Delta3D Internal Applications are 
rated It for Eicensing. 

With respect to Innovation, Delta3D Internal Applications do not have a 
community collaborating or sharing with them nor is there anything built-in to facilitate 
this. These are powerful and useful tools provided by the Delta3D development team but 
are final products themselves—not something with which to innovate a new solution. In- 
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tegrating, Extending, or Modifying DeltaSD Internal Applieations are rated IIIi for Inno¬ 
vation. 

DMZ 

DMZ Internal Applieations eonsist of a few seripts and the Imk build sys¬ 
tem. Integrating with these tools is as simple as eommand line ealls. Therefore, Integrating 
with DMZ Internal Applieations is rated Ij for Standards. 

Other than editing the seripts or the Imk build system, there is no method 
for Extending or Modifying DMZ Internal Applieations. Again, the eode is open souree, 
but that is a different issue, and sinee these Internal Applieations are not themselves built 
with DMZ, whieh does provide a standard meehanism for ehanging behavior. Extending or 
Modifying DMZ Internal Applieations are rated Illg for Standards. 

With respeet to Eieensing, the DMZ Internal Applieations are released un¬ 
der the MIT lieense. As sueh developers are not restrieted in their use or redistribution. 
Integrating, Extending, or Modifying DMZ Internal Applieations are rated 1^ for Eieens- 
ing. 

With respeet to Innovation DMZ Internal Applieations do not have a eom- 
munity eollaborating or sharing with them nor is there anything built-in to faeilitate this. 
These are seripts to aid in DMZ development, not something with whieh to innovate a new 
solution. Integrating, Extending, or Modifying DMZ Internal Applieations are rated Hit 
for Innovation. 


c. Middleware 

The Middleware is the “meat” of the frameworks. These are the libraries, 
elasses, plugins, and more that developers use in building their simulations. It is the Mid¬ 
dleware that perhaps has the greatest influenee on the day to day development of a simula¬ 
tion. 

Delta3D 

With respeet to Standards, DeltaSD Middleware ean be Integrated through 
standard C-i-i- funetion ealls through published headers. The meehanism for integrating 
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with with the libraries is open, doeumented, and has some eommunity partieipation. Inte¬ 
grating DeltaSD Middleware is rated Is for Standards. 

Extending DeltaSD Middleware eould involve something as simple as sub- 
elassing a library elass and using the new ehild elass in plaee of the parent, or it eould re¬ 
quire ehanges to the original souree eode, as with Internal Applieations. Extending DeltaSD 
Middleware is rated Ilg for Standards. 

There is no approved teehnique for Modifying the DeltaSD Middleware 
aside from editing the souree eode. Therefore, Modifying DeltaSD Middleware is rated 
Ills for Standards. 

With respeet to Eieensing, DeltaSD Middleware is released under the EGPE. 
As sueh, developers are not restrieted in their use or redistribution. Integrating, Extending, 
or Modifying DeltaSD Middleware is rated I; for Eieensing. 

With respeet to Innovation, DeltaSD Middleware has some eommunity in¬ 
volvement for eollaboration and sharing, but the software is not partieularly suited to users 
making unexpeeted and innovative uses of other people’s work. Some sueh sharing goes on 
within the halls of NPS as students share eode, but this does not aehieve high innovation. 
Integrating DeltaSD Middleware is rated Hi for Innovation. 

Trying to Extend or Modify DeltaSD Middleware is even less friendly to 
Maxwell’s (2006) eoneept of Innovation. Extending and Modifying DeltaSD Middleware 
is rated I Hi for Innovation. 

DMZ 

With respeet to Standards, DMZ Middleware ean be Integrated through the 
standardized ealls to the Objeet module or Event module, and these ealls ean be made with 
C-i-i- or JavaSeript. Eurther integration is possible with simple eonfiguration of the XME 
files that define an applieation. Integrating DMZ Middleware is rated Ig for Standards. 

Extending or Modifying DMZ middleware is possible through the same 
meehanism by whieh one Integrates. This is an arehiteetural benefit of having loosely 
eoupled eomponents that eommunieate through restrieted meehanisms, namely the Objeet 
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and Event modules. The XML eonfiguration files permit easy swapping out of eomponents 
making Modifying as normal an operation as Integrating, possibly more so. Extending and 
Modifying DMZ Middleware is also rated Ig for Standards. 

With respeet to Lieensing, DMZ Middleware is released under the MIT li- 
eense. As sueh, developers are not restrieted in their use or redistribution. Integrating, 
Extending, or Modifying DMZ Middleware is rated Ij, for Lieensing. 

With respeet to Innovation, DMZ Middleware Integration is well-suited to a 
eollaborative and sharing environment. Component purpose and funetion are isolated, and 
there are simple, standardized teehniques for eomposing applieations. To date, there is not 
a DMZ eommunity to speak of, exeept the development team that built and uses DMZ, and 
here power of innovation is used to good effeet as one person’s eomponents are eomposed 
into another’s projeet. Although the promise for good innovation is present, there is little 
eommunity for proof. Integrating DMZ Middleware is rated lit for Innovation. 

Extending and Modifying DMZ Middleware enjoys the same promise of In¬ 
novation, and some proof of this is borne out in its development team. Component behavior 
ean be Extended or Modified using the same teehnique as Integration, but as with Integra¬ 
tion, there is little eommunity for proof. Extending and Modifying DMZ Middleware is 
rated lit for Innovation. 

d. Kernel 

The Kernel is the portion of the framework that “makes it tiek.” Developers 
do not generally need aeeess to the Kernel for defining their simulations, though they likely 
will aeeess it in some way in order to run their simulations. 

Delta3D 

With respeet to Standards, the DeltaSD Kernel ean be Integrated through 
standard C-i-i- funetion ealls through published headers. As with the Middleware, the meeh- 
anism for integrating with the kernel is open, doeumented, and has some eommunity par- 
tieipation. Integrating the DeltaSD Kernel is rated Ig for Standards. 
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Extending the DeltaSD Kernel ean be aehieved easily through inheritanee or 
with more diffieulty through editing souree eode. There are no standardized teehniques or 
arehiteetural allowanees for enhaneing the behavior of the Kernel. Extending the DeltaSD 
Kernel is rated IIs for Standards. 

There is no approved teehnique for Modifying the DeltaSD Kernel aside 
from editing the souree eode. Therefore, Modifying the DeltaSD Kernel is rated Ills for 
Standards. 

With respeet to Eieensing the DeltaSD Kernel is released under the EGPE. 

As sueh, developers are not restrieted in their use or redistribution. Integrating, Extending, 
or Modifying DMZ Middleware is rated I; for Eieensing. 

With respeet to Innovation, the DeltaSD Kernel is situated in a similar way 
to the Middleware whieh, although it is possible to eollaborate and share innovative so¬ 
lutions that Integrate with the Kernel, there is little eommunity taking advantage of it. 
Integrating the DeltaSD Kernel is rated lit for Innovation. 

Extending and Modifying the DeltaSD Kernel is not well suited to innova¬ 
tion, and there is no eommunity of aetivity there. Extending and Modifying the DeltaSD 
Kernel is rated Hit for Innovation. 

DMZ 

With respeet to Standards, the DMZ Kernel ean be Integrated through stan¬ 
dard C-i-i- funetion ealls through published headers like DeltaSD, but unlike the rest of 
DMZ, the Kernel is not itself a eomponent arehiteeture (although some parts of the foundation 
direetory are minor eomponents). The DMZ Kernel is not well-doeumented, and this im¬ 
pairs a developer’s ability to use it. It is instead the meehanism that loads eomponents and 
enables their intereonneetions. Integrating the DMZ Kernel is rated Ilg for Standards. 

Extending or modifying the DMZ Kernel ean only be aeeomplished through 
editing the souree eode. There is little room even for extension by inheritanee. Extending 
and Modifying the DMZ Kernel is rated Ills for Standards. 
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With respect to Licensing, the DMZ Kernel is released under the MIT li¬ 
cense. As such developers are not restricted in their use or redistribution. Integrating, 
Extending, or Modifying DMZ Middleware is rated 1^ for Licensing. 

With respect to Innovation, the DMZ Kernel can enjoy collaboration and 
sharing of innovative ways to Integrate the Kernel, but there is little community taking 
advantage of it. Integrating the DMZ Kernel is rated lit . 

Extending and Modifying the DMZ Kernel does not enjoy the same benefits 
as its Middleware, which is component based. Changing the Kernel requires editing its 
source code, and there is no community activity there. Extending and Modifying the DMZ 
Kernel is rated lilt for Innovation. 

6. Results 

We gain insight by applying the criteria, and the results of the openness model 
applied to these two frameworks is summarized in Table 10. Although this example only 
shows us two frameworks of the many that may be assessed in the future, it demonstrates 
the usefulness of the model’s contribution and its success in being able to differentiate 
between two visual simulation frameworks—two open source frameworks at that. Some 
observations may be made regarding the model. 

Assigning notional weights to the weighting function allows for further analysis. 
These weights would be customized according to the needs of the program manager. As¬ 
sume a weight of 1 for 1,2 for II, and 3 for III (Equation 2). 




1, if Rioi = I 

2, ifRtoi = II , Vl,o,i 

3, ifRtoi^III 


( 2 ) 


With a set of possible weights provided and focusing on the development opera¬ 
tions, we can plot how the model differentiates between frameworks. Eigures 7 and 8 show 
the results of plotting the weighted values as a stacked bar chart. 
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Table 10: Because Openness is more than licensing, these two example open source simu¬ 
lation frameworks are not equal with respect to openness. 


Layer 

Operation 

DeltaSD 


DMZ 

Std 

Lie 

Inn 

Std 

Lie 

Inn 


Integrate 

IIs 

III 

nil 

Is 

III 

Hi 


Extend 

IIs 

III 

nil 

Is 

III 

Hi 

Applications 

Modify 

IIs 

III 

nil 

Is 

III 

Hi 


Integrate 

Is 

II 

nil 

Is 

Ii 

nil 


Extend 

Ills 


nil 

Ills 

It 

nil 

Applications 

Modify 

Ills 

II 

iii: 

Ills 

Ii 

in: 


Integrate 

Is 

II 

III 

Is 

Ii 

Hi 

Middleware 

Extend 

IIs 

II 

nil 

Is 

Ii 

Hi 


Modify 

Ills 

II 

nil 

Is 

Ii 

Hi 


Integrate 

Is 

II 

Hi 

IIs 

Ii 

Hi 

Kernel 

Extend 

IIs 

II 

Illi 

Ills 

Ii 

nil 


Modify 

Ills 

II 

nil 

Ills 

Ii 

nil 


Std, s = Standards; Lie, I = Licensing; Inn, i = Innovation; 
I, II, III are classifications. 


Comparison of Weighted Ratings for Standards 


External Applications 
Internal Applications 
Middleware 
Kernel 


Delta3D DMZ Delta3D DMZ Delta3D DMZ 
Integrate Extend Modify 


Figure 7: The two frameworks differ with respect to standards and development operation. 
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Comparison of Weighted Ratings for Innovation 


External Applications 
Internal Applications 
Middleware 
Kernel 


Delta3D DMZ Delta3D DMZ Delta3D DMZ 
Integrate Extend Modify 


Figure 8: The two frameworks differ with respect to innovation and development operation. 


7. Insights Gained 

Not only does the model differentiate between the two frameworks, but the pro¬ 
cess of applying the model reveals insights. These insights, some examples of which are 
presented below, are an additional benefit of the model. 

a. External Applications 

One might remark that the External Applications section looks rather “bor¬ 
ing” or has “low variability.” This reminds us that External Applications are built by third 
parties, and that because of the latitude afforded by the licensing, developers using either 
framework may or may not generate open applications. 

That the architecture of DMZ permits even third party applications to be 
manipulated in a standard fashion is a benefit that should not be overlooked. The govern¬ 
ment should be pleased that code that it pays for is open and accessible for use by other 
agencies. 

b. Uniformity Across Operations 

Although DeltaSD has some variability across the operations Integrate, Ex¬ 
tend, and Modify, DMZ shows little differentiation, especially in Middleware. This high¬ 
lights a potential advantage of the component architecture in that it makes operations on 
the code more uniform. This should be seen as a big win for DMZ and possibly component 
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architectures in general. On the other hand DeltaSD has consistency between Middleware 
and Kernel, whereas DMZ’s Kernel is entirely different from its Middleware. Depending 
on the needs of the project, one may win out over the other. 

c. Open Source License 

It might come as a surprise to see a Ilj^ rating for both frameworks for 
External Applications since both frameworks are open source, but this highlights an inter¬ 
esting point about licensing. The licenses used for these frameworks do not require that 
External Applications be released as open source software. Erom one person’s point of 
view, this may be a negative, because the work that a third party develops will not be ac¬ 
cessible. Erom that third party’s point of view, this may be desirable, because they are 
not forced to release their source code. The GPE is the classic example of a “viral” open 
source license that forces developers to release subsequent software as GPE. This would 
have been rated Ij, because of its strong insistence on open licensing, but whether or not it 
is a “good” thing is subjective. 

C. SUMMARY 

We have successfully shown that our openness model differentiates between two 
visual simulation frameworks and that we gain valuable insight in the process of applying 
and interpreting the model. The model contributes a new approach and tool for program 
managers and others to assess the nature of visual simulation frameworks with respect to 
openness. 

Breaking out openness into more than just licensing, which is often what people 
think of when they hear “open,” aided in the differentiation between frameworks. The 
standards issue in particular revealed architectural differences between the two frameworks 
we tested. Breaking out the operations revealed a valuable uniformity in interacting with 
one of the frameworks. 
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Weighting the categories according to layer, operation, and issue allows for cus¬ 
tomizing the model according to the values of a particular user. This also provides a nu¬ 
merical summary from the categorical data and may aid in analysis and decision support. 
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IV. REUSE 


In this chapter, we develop and apply a model for assessing reuse based on the def¬ 
inition and need established in Chapter II: Literature Review. The model uses established 
software metrics and applies them in a new way in order to differentiate visual simulation 
frameworks based on their potential for reuse. These metrics are acknowledged to relate 
not only to reusability but also to general quality as well. We conduct a study in which we 
show that the model differentiates between two visual simulation frameworks. 

A. DEVELOPING THE REUSE MODEL 

We learned from Chapter II: Literature Review that there are internal and external 
attributes that affect reusability. Some external attributes that affect reusability are com¬ 
prehensibility and maintainability. Some internal attributes that are related to these are 
complexity, coupling, and cohesion. We select relevant metrics for complexity, coupling, 
and cohesion and determine criteria for estimating transition points in the values of those 
metrics. 

1. Complexity, Coupling, and Cohesion Metrics 

Two major complexity metrics for procedural programming, McCabe (1976) and 
Halstead (1977), have survived as viable techniques for estimating complexity. The McCabe 
technique is still used in estimating complexity of methods within a class. This is used 
sometimes in the CK metric WMC, which is discussed in this chapter. Both techniques are 
often available in software development tools available to programmers. 

McCabe (1976) describes a graph-theoretic, lexical, complexity measure that in¬ 
spects the potential flow of execution through a program. Changes to a program flow re¬ 
sulting from such structures as if statements and for loops add to the complexity count. 
This metric is based on the decision structure of a program and is independent of the size, 
e.g., adding a function does not increase the potential paths of execution. 
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McCabe defined the metric in terms of graph theory. The cyclomatic number v(G) 
of a graph G with n vertices, e edges, and p connected components is 

v(G) = e —n + p. (3) 

The examples given in Figure 9 help illustrate how the calculations are affected 
by some common control structures. In practice one often counts the predicates that are 
directly observable in source code and adds one, which McCabe proves to be equivalent to 
the graph-theoretic formulation. 



Sequence If Then Else While Until 

v=1-2+2=1 v=4-4+2=2 v=3-3+2=2 v=3-3+2=2 

Figure 9: McCabe’s cyclomatic complexity measures the possible paths of execution 
through a program. 


McCabe found evidence that code with higher complexity values was less reliable 
and more “troublesome.” 

The CK suite of metrics are grounded in theory, relevant to 00 programming, and 
validated empirically. Because of the importance of class design (Champeaux, Lea, & 
Faure, 1992), the CK metrics focus on measuring the complexity in the design of classes. 

a. Weighted Methods per Class 

Weighted Methods per Class (WMC) is a weighted sum of the count of 
methods in a class. S. R. Chidamber and Kemerer intentionally do not require how to 
weight the methods, suggesting that one could use a more traditional (procedural) com¬ 
plexity metric but that it is best left as an implementation decision by the practitioner. Li 
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and Henry (1993a) assume the use of McCabe’s cyclomatic complexity. With a weight of 
1 for all methods, the metric is simply the number of methods in each class. 

To calculate WMC for a class A, there are at least two methods: 

• Sum the chosen complexity metric for each method in class A. If mAi is method 1 of 
class A with n methods and |mAtl is the chosen complexity metric for the method, 
then 


WMC(A) = _^|mAi|. (4) 

1=0 

• Assume complexity of each method is the same, and count the number of methods 
in A. This technique is popular because of its simpler implementation and because 
there is disagreement over which complexity metric ought to be used. 

They make several observations: 

• The number of methods and the complexity of methods involved is a predictor of 
how much time and effort is required to develop and maintain the class. 

• The larger the number of methods in a class the greater the potential impact on chil¬ 
dren, since children will inherit all the methods defined in the class. 

• Classes with large numbers of methods are likely to be more application specific, 
limiting the possibility of reuse. 

b. Depth of Inheritance Tree 

Depth of Inheritance Tree (DIT) relates to scope and is a measure of how 
many ancestor classes could potentially affect this class. 

To calculate DIT for class A one counts the number of parent classes one 
must traverse to reach the root. A root object such as java . lang .Object would have 
DIT = 1. 
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They make several observations: 


• The deeper a class is in the hierarchy, the greater the number of methods it is likely 
to inherit, making it more complex to predict its behavior. 

• Deeper trees constitute greater design complexity, since more methods and classes 
are involved. 

• The deeper a particular class is in the hierarchy, the greater the potential reuse of 
inherited methods. 

c. Number of Children 

Number of Children (NOC) is the number of immediate subclasses, which 
also relates to scope. 

To calculate NOC for class A one counts the number of classes which di¬ 
rectly inherit from A. 

They make several observations: 

• The greater the number of children, the greater the reuse, since inheritance is a form 
of reuse. 

• The greater the number of children, the greater the likelihood of improper abstraction 
of the parent class. If a class has a large number of children, it may be a case of 
misuse of subclassing. 

• The number of children gives an idea of the potential influence a class has on the 
design. If a class has a large number of children, it may require more testing of the 
methods in that class. 

d. Coupling between Objects 

Coupling Between Object Classes (CBO) relates to the interconnectedness 
of otherwise-unrelated (through inheritance) classes. 
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To calculate CBOfor class A one counts the number of classes, beside A or 
ancestors of A, which are referenced either by instance variable or method call. 

They make several observations: 

• Excessive coupling between object classes is detrimental to modular design and pre¬ 
vents reuse. The more independent a class is, the easier it is to reuse it in another 
application. 

• In order to improve modularity and promote encapsulation, inter-object class couples 
should be kept to a minimum. The larger the number of couples, the higher the 
sensitivity to changes in other parts of the design, and therefore, maintenance is more 
difficult. 

• A measure of coupling is useful to determine how complex the testing of various 
parts of a design are likely to be. The higher the inter-object class coupling, the more 
rigorous the testing needs to be. 

The CBO metric has been the basis for many other coupling metrics. It is 
sometimes criticized (Hitz & Montazeri, 1995; Kitchenham, 2010) yet remains in use and 
validated by others (Basili et ah, 1996; S. Chidamber et ah, 1998). 

e. Response for a Class 

Response for a Class (RFC) is the number of methods that could possibly 
be called in response to a message being received. 

To calculate RFC for class A one counts the number of methods in A and 
the number of methods that are called from within methods of A. 

S. R. Chidamber and Kemerer and Li and Henry (1993a) do not specify 
whether or not this includes or excludes methods called to outside classes, but Hitz and 
Montazeri (1995) clarify RFC as the “union of the protocol a class offers to its clients and 
the protocols it requests from other classes.” 

S. R. Chidamber and Kemerer make several observations: 
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• If a large number of methods can be invoked in response to a message, the testing and 
debugging of the class becomes more complicated since it requires a greater level of 
understanding required on the part of the tester. 

• The larger the number of methods that can be invoked from a class, the greater the 
complexity of the class. 

• A worst case value for possible responses will assist in appropriate allocation of 
testing time. 

/. Lack of Cohesion in Methods 

Lack of Cohesion in Methods (LCOM) relates to the degree of similarity, 
the difference among the instance variables used by each method in a class. This helps 
identify classes that may be trying to do too many things and whose behavior will be less 
predictable. 

To calculate LCOM for class A, there are two similar but incompatible tech¬ 
niques that people use: 

• Count the number of method pairs in A that do not use any of the same instance 
variables and subtract the number of method pairs in A that have at least one instance 
variable in common. Negative values are often reported as zero (Basil! et ah, 1996). 

• Calculate the percentage of method pairs that do not use any of the same instance 
variables. This technique normalizes for the size of the class. Unfortunately the 
empirical validations that have been performed do not use this technique (Section 2) 

They make several observations: 

• Cohesiveness of methods within a class is desirable, since it promotes encapsulation. 

• Lack of cohesion implies classes should probably be split into two or more sub¬ 
classes. 
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• Any measure of disparateness of methods helps identify flaws in the design of classes. 

• Low cohesion increases complexity, thereby increasing the likelihood of errors dur¬ 
ing the development process. 

These CK metrics have been examined, criticized, praised, and embraced 
by many researchers and practitioners, and are the metrics used in this research. However, 
for completeness, what follows are some other interesting metrics and research that has 
been done to improve the CK metrics. As so often happens with finer- and finer-grained 
improvements, the community that accepts them shrinks as well, and the original stands 
the test of time. 

2. Empirical Evaluation of Metrics 

Researchers have conducted studies to validate the CK metrics. These validations 
provide insight into what values are considered expected for the metrics although no liter¬ 
ature discovered provided a thorough review of all empirical studies in order to generate 
consensus on these levels. The studies used metrics to predict fault-proneness as a sur¬ 
rogate for estimating software quality. Some literature reminds practitioners to use the 
metrics to guide further analysis when results are unexpected, and the same advice applies 
to the acquisition professional assessing simulation frameworks. 

The empirical studies discovered in the literature cautioned users that no metric 
was a perfect predictor. However, these studies also confirmed that there was value in 
evaluating software against these metrics. For studies with significant findings for given 
metrics, a transition point from better to worse was calculated as one standard deviation 
above the mean. 

Li and Henry (1993b) evaluated five of the CK metrics (not CBO) and their own 
metrics Message Passage Coupling (MPC) and Data Abstraction Coupling (DAC) to pre¬ 
dict maintainability in a study of two commercial Ada applications built with Classic- 
Ada.'^'^ They calculated WMC using the McCabe cyclomatic complexity technique, unlike 
the following studies which used the equal weighting method. Since the WMC metric dif- 
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fers in kind from the remaining studies, it is not reported here. Transition points for four 
metrics were calculated from the statistical results: DIT ^ 2.7, NOC ^ 1.6, RFC ^ 75.6, 
LCOM ^ 15.5. 

Basil! et al. (1996) evaluated eight information management systems against all 
six CK metrics. They found that the correlations among the metrics were weak, and that 
the relationship between CBO and RFC were most independent. They found that LCOM 
values were near zero in their systems and so did not provide any meaningful differentiation 
for their evaluation. Larger NOC values were better in their systems. Transition points for 
four metrics were calculated from the statistics results: WMC ^ 28.3, DIT ^ 3.3, CBO ^ 
14.4, RFC ^ 67.3. 

S. Chidamber et al. (1998) evaluated three software systems of an unnamed Euro¬ 
pean bank. They found correlation among WMC, RFC, and CBO and suggest that subsets 
of the six CK metrics may be sufficient in assessing software. Transition points for six 
metrics were calculated from the statistical results: WMC ^ 19.6, DIT ^ 1.5, NOC ^ 1.4, 
CBO ^ 9.4, RFC ^ 37.6, and LCOM ^ 54.8. 

Tang et al. (1999) evaluated a supervisory control and data acquisition system con¬ 
sisting of over 200 subsystems and 3 million lines of C-t-i- code. They measured the six 
CK metrics and only validated WMC and RFC. Transition points for two metrics were 
calculated from the statistical results: WMC ^19.6 and RFC ^ 100.2. 

Cartwright and Shepperd (2000) evaluated a system from a large European telecom¬ 
munications company. The company employed more than 2,000 developers, and the system 
evaluated consisted of 133,000 lines of C-t-i- source code. Of the CK metrics, they were 
only able to calculate DIT and NOC and found that the maximum values encountered were 
two and four respectively. Standard deviations were not reported. Because of the low val¬ 
ues for these metrics, they only offered subjective evaluations but noted that the two CK 
metrics that they used could point out problematic classes. 

Olague et al. (2007) evaluated the open source Mozilla project Rhino, a 100% Java 
implementation of JavaScript. Rhino is considered an example of agile software develop- 
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ment based on the principles of the Agile Alliance (Olague et ah, 2007; Agile Alliance, 
2011). They evaluated six releases of the source code from vl.4R3 to vl.5R5, which con¬ 
sists of 148 classes and 36,000 lines of code. They used an alternate cohesion metric, so 
their results are not comparable here. Transition points for five metrics were calculated 
from the statistical results: WMC ^ 28.9, DIT ^ 1.4, NOC ^ 1.5, CBO ^ 6.4, and RFC 
^ 32.7. 

These empirical studies can be summarized to form a loose understanding of transi¬ 
tion points from better to worse values for metrics (Table 11). These values should be taken 
with caution. Since our purpose for them is to provide insight into assessing software, not 
to predict hard outcomes, they still are useful. 

Table 11: Transition points identified in empirical validation studies in the literature. 


Paper 

WMC 

DIT 

NOC 

CBO 

RFC 

Li & Henry, 1993b 

- 

2.7 

1.6 

- 

75.6 

Basil! et ah, 1996 

28.3 

3.3 

- 

14.4 

67.3 

S. Chidamber et ah, 1998 

19.6 

1.5 

1.4 

9.4 

37.6 

Tang et ah, 1999 

19.6 

- 

- 

- 

100.2 

Cartwright & Shepperd, 2000 

- 

2.0 

4.0 

- 

- 

Olague et ah, 2007 

28.9 

1.4 

1.5 

6.4 

32.7 


Boldface indicates chosen transition points. 


There is some variation among the values observed in the validation experiments, 
and this seems to be accepted in the literature. Therefore, it is only through our summariz¬ 
ing the body of studies that transition points are identified. 

3. Criteria for Metrics 

There is no consensus in the literature for precise divisions in the metrics regarding 
“good” and “bad” values. However, ranges can be inferred from normal to higher than 
normal by examining the statistical results of several empirical studies and adding one 
standard deviation to the mean. This is chosen as a conservative estimator. The one sided 
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Tchebysheff inequality (Equation 5) tells us that the mean plus one standard deviation 
will always eontain both the mean and median, regardless of the underlying distribution 
(Equation 6). Sinee too little data is reported in the literature to provide a more aggressive 
dividing line, p. plus k = 1 standard deviations is a reasonable ehoiee. Three levels are 
defined here for the six CK metries. These levels are based on the low and high estimates 
diseovered in empirieal studies. 


Pr(X - p. ^ kcr) ^ 

Ip. — m| ^ cr (6) 

a. Weighted Method Complexity 

When using equal eomplexity weights among methods (eounting the meth¬ 
ods in a class), the lowest published level that we infer as a transition is 19.6 (S. Chidamber 
et ah, 1998; Tang et ah, 1999). The highest level is 28.9 (Olague et ah, 2007). Therefore, 
we divide the criteria into three categories: Iwmc for WMC = [0,19.6), IIwmc for 
WMC = [19.6,28.9), and IIIwmc for WMC = [28.9,oo). 

b. Depth of Inheritance Tree 

Empirical studies found that many projects had very low values of DIT. 
The lowest published level that we infer as a transition is 1.4 (Olague et ah, 2007), and the 
highest level is 3.3 (Basili et ah, 1996). We divide the metric into three categories: Idit 
for DIT = [0,1.4), IIdit for DIT = [1.4,3.3), and IIIdit for DIT = [3.3,oo). 

c. Number of Children 

Empirical studies suggested that a high number of children could indicate 
unnecessary inheritance and thus induce inheritance coupling. The lowest published level 
that we infer as a transition is 1.5 (S. Chidamber et ah, 1998), and the highest level is 4.0 
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(Cartwright & Shepperd, 2000). We divide the metrie into three eategories: Inoc for 
NOC = [0,1.4), IInoc for N0C = [1.4,4.0), and IIInoc for NOC = [4.0,oo). 

d. Coupling between Objects 

Although there have been many follow-on metries related to eoupling, CBO 
remains a meaningful metrie. The lowest published level that we infer as a transition is 
6.4 (Olague et ah, 2007), and the highest level is 14.4 (Basil! et ah, 1996). We divide the 
metrie into three eategories: Icbo for CBO = [0,6.4), IIcbo for CBO = [6.4,14.4), and 
IIIcBO for CBO = [14.4,00). 

e. Response Set for a Class 

Studies eonfirmed the effeetiveness of the RFC metrie. The lowest pub¬ 
lished level that we infer as a transition is 32.7 (Olague et ah, 2007), and the highest level 
is 100.2 (Tang et ah, 1999). We divide the metrie into three eategories: Irfc for RFC = 
[0,32.7), IIrfc for RFC = [32.7,100.2), and IIIrfc for RFC = [100.2,oo). 

/. Lack of Cohesion in Methods 

There have also been many follow-on metries to LCOM, but it stands as a 
useful metrie. Unfortunately the version of LCOM that is reported in the studies is not 
the pereentage method, whieh is normalized for elass size, but rather the eount of method 
pairs that do not share a eommon instanee variable minus the eount of method pairs that 
do. Beeause we do not have published LCOM values that we ean apply, we strike LCOM 
from our list. 


g. Summary of Reuse Criteria 

Criteria for these metries vary in the literature, but they ean be sifted and 
summarized to provide some level of insight into the software being assessed. Table 12 
summarizes the eriteria used in this researeh to represent three quality levels. 
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Table 12: A thorough review of empirical validation studies provides transition points to 
use as criteria with software metrics. 


Metric 

Lower Bounds 

I 

II 

III 

Weighted Methods per Class (WMC) 

0 

19.6 

28.9 

Depth of Inheritance Tree (DIT) 

0 

1.4 

3.3 

Number of Children (NOC) 

0 

1.4 

4.0 

Coupling Between Object Classes (CBO) 

0 

6.4 

14.4 

Response for a Class (RFC) 

0 

32.7 

100.2 


4. Model for Assessing Reuse 

To apply the model to the software frameworks, we calculate the metrics using a 
software analysis tool and apply the criteria to populate the model. 

For continuity with the openness model and to aid in understanding different por¬ 
tions of the code, the criteria are applied to the two layers Middleware and Kernel. It should 
be expected that External and Internal Applications are assessed, since they represent sim¬ 
ulations built or tools included with the framework, not the framework itself. Clear lines 
must be drawn with each framework regarding which code belongs in which layer. 

The data may be presented such as in Table 13 where all of the ratings are presented 
at once, and both a detailed inspection and a high-level glance can provide insight. 

5. Weights for User Assigned Value Systems 

If a final numeric score is desired, weights can be assigned to the categories accord¬ 
ing to what layer and metric are most important. When summed, these weights provide for 
a tailored assessment of the frameworks. 

Let L be the set of layers L = {Middleware, Kernel} and I G L be a layer. Let M be 
the set of metrics (WMC, DIT, NOC, CBO, RFC} and m G M be a metric. Let Rj^Tn, be the 
categorical rating assigned to a framework at the given layer and metric. Let wi,Ta(Rim) 
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Table 13: A model for presenting metrics for simulation frameworks help identify potential 
risks to reusability. 


Code 


Classes WMC DIT NOC CBO RFC 


DeltaSD 


x,xxx I 


Middleware 

Kernel 


II 

III 


DMZ 


Middleware 

Kernel 


be the weighting function that returns the user assigned value for a given R;ra- Then the 
overall openness value Vr of a framework is given by Equation 7: 



(7) 


leL meM 


The development of these weights for a particular use case is beyond the scope of 
this dissertation, but they may be used to help score frameworks against the specific needs 
of a program manager. 

B. STUDY 2: ASSESSING REUSE 

To demonstrate the feasibility of this assessment model, two simulation frameworks 
DeltaSD and DMZ (described in Sections 2 and 3) were analyzed with the reuse model. To 
calculate the metrics, we used Understand from Scientific Toolworks, Inc. This software 
has a Perl API for accessing the lexical database created by a software project. Through 
this API one can write scripts to collect metrics that are not built-in to the software itself, 
such as the scripts used to collect CK metrics here. 
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1. Methodology 

The first step in the reuse study was loading the two frameworks into the analysis 
tool Understand. This tool parsed all the souree files in the two projeets and ereated a 
lexieal database that eould be queried later. 

A number of shell, awk, and Perl seripts (Appendix A) were used to extraet the nee- 
essary metries. Some metries were generated by the reporting funetion within Understand 
itself, and some required aeeessing the lexieal database with the API through Perl. 

The metries were eompiled in spreadsheets and keyed by elass or filename, as ap¬ 
propriate. From the spreadsheets, the data eould be aggregated by framework, layer, or 
other division. The ratings were assigned within the spreadsheet by formulas based on the 
transition points developed from the literature. 

2. Calculating the Metrics 

The two eodebases were analyzed to ealeulate CK metries. The data was aggre¬ 
gated by namespaee for DeltaSD and major direetory for DMZ and then aggregated again 
aeeording to the software layers defined earlier: Middleware and Kernel. A detailed listing 
of the results is given in Table 14 for DeltaSD and Table 15 for DMZ. A summary at the 
Middleware and Kernel levels is given in Table 16. 

3. Applying the Criteria 

We ean now apply the eriteria provided in the previous ehapter to these metries. 
Eaeh metrie value is eompared against the transition points to determine the rating. The 
data was aggregated by namespaee for DeltaSD and major direetory for DMZ and then 
aggregated again aeeording to the software layers defined earlier: Middleware and Kernel. 
A detailed listing of the resulting ratings is given in Table 17 for DeltaSD and Table 18 
for DMZ. A summary is given in Table 19. Subseripts on the eategories, e.g., W1\AC in 
IwMc . are left off to improve readability in the densely-paeked tables. 
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Table 14: Detailed listing of CK metrics for the DeltaSD framework 


Code 

Classes 

WMC 

DIT 

NOC 

CBO 

RFC 

DeltaSD 

1,191 

12.43 

1.96 

0.63 

7.43 

38.84 

Middleware 

822 

11.51 

1.86 

0.49 

7.32 

39.64 

dtABC 

31 

14.58 

2.77 

0.42 

9.45 

56.42 

dtActors 

98 

9.72 

4.39 

0.30 

7.68 

102.63 

dtAI 

97 

10.81 

0.93 

0.33 

5.33 

18.09 

dtAnim 

55 

10.91 

1.60 

0.15 

9.11 

29.22 

dtAudio 

14 

18.86 

3.07 

0.00 

7.79 

67.79 

dtDirector 

98 

15.93 

2.51 

0.60 

13.20 

65.09 

dtDIS 

16 

7.81 

0.69 

0.13 

10.25 

10.25 

dtGUI 

19 

14.79 

1.32 

0.11 

9.74 

22.37 

dtHLAGM 

46 

13.67 

1.13 

0.24 

6.43 

23.93 

dtInputlSense 

1 

11.00 

4.00 

0.00 

7.00 

59.00 

dtInputPLIB 

1 

12.00 

4.00 

0.00 

6.00 

60.00 

dtInspectorQt 

21 

16.71 

1.95 

0.86 

9.14 

22.10 

dtLMS 

13 

7.31 

1.00 

0.00 

4.15 

18.85 

dtNet 

3 

12.67 

1.67 

0.00 

13.67 

19.67 

dtNetGM 

11 

16.82 

2.27 

0.18 

11.91 

42.55 

dtQt 

46 

19.17 

1.63 

0.46 

13.59 

49.22 

dtScript 

1 

11.00 

3.00 

0.00 

4.00 

32.00 

dtTerrain 

54 

7.83 

1.39 

0.11 

5.22 

22.41 

dtUtil 

96 

11.07 

0.66 

1.64 

3.43 

14.56 

NA 

57 

3.82 

1.56 

0.32 

2.07 

29.93 

psGeodeTransform 

1 

1.00 

1.00 

0.00 

6.00 

1.00 

sigslot 

43 

6.00 

1.30 

0.65 

2.51 

9.95 

Kernel 

369 

14.48 

2.20 

0.93 

7.69 

37.08 

dtCore 

155 

20.70 

2.15 

1.01 

8.80 

45.49 

dtDAL 

111 

8.95 

2.61 

1.03 

7.30 

30.14 

dtGame 

103 

11.10 

1.84 

0.69 

6.46 

31.90 
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Table 15: Detailed listing of CK metrics for the DMZ framework 


Code 

Classes 

WMC 

DIT 

NOC 

CBO 

RFC 

DMZ 

475 

13.39 

0.92 

1.19 

8.37 

41.81 

Middleware 

319 

13.52 

1.17 

0.77 

10.24 

52.82 

frameworks/archive 

11 

15.91 

1.18 

0.82 

10.82 

53.73 

frameworks/audio 

11 

15.18 

0.82 

0.27 

10.00 

42.64 

frameworks/entity 

28 

12.96 

1.89 

0.04 

11.39 

138.00 

frameworks/event 

10 

23.20 

0.60 

1.20 

9.50 

42.70 

frameworks/input 

18 

16.56 

1.00 

2.11 

10.06 

43.11 

frameworks/net 

71 

6.08 

1.07 

0.80 

7.17 

19.08 

frameworks/object 

27 

20.04 

1.26 

2.96 

10.00 

73.67 

frameworks/qt 

80 

15.88 

1.24 

0.30 

11.45 

52.75 

frameworks/render 

57 

13.19 

0.95 

0.37 

11.19 

42.51 

frameworks/weapon 

6 

14.00 

1.67 

0.00 

18.33 

122.67 

Kernel 

156 

13.12 

0.43 

2.06 

4.54 

19.31 

foundation/libs 

26 

11.73 

0.42 

0.38 

3.19 

16.85 

foundation/plugins 

10 

9.40 

1.00 

0.00 

7.90 

31.40 

kernel/runtime 

70 

11.29 

0.53 

4.06 

6.51 

17.69 

kernel/system 

19 

15.21 

0.37 

1.32 

1.47 

23.05 

kernel/types 

31 

18.32 

0.06 

0.10 

2.03 

18.84 


Table 16: Summary listing of CK metrics for DeltaSD and DMZ frameworks 


Code 

Classes 

WMC 

DIT 

NOC 

CBO 

RFC 

Delta3D 

1,191 

12.43 

1.96 

0.63 

7.43 

38.84 

Middleware 

822 

11.51 

1.86 

0.49 

7.32 

39.64 

Kernel 

369 

14.48 

2.20 

0.93 

7.69 

37.08 

DMZ 

475 

13.39 

0.92 

1.19 

8.37 

41.81 

Middleware 

319 

13.52 

1.17 

0.77 

10.24 

52.82 

Kernel 

156 

13.12 

0.43 

2.06 

4.54 

19.31 
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Table 17: Detailed listing of reuse ratings for the DeltaSD framework 


Code Classes WMC DIT NOC CBO RFC 



DeltaSD 


1,191 


Middleware 


822 


dtABC 

dtAetors 

dtAI 

dtAnim 

dtAudio 

dtDireetor 

dtDIS 

dtGUI 

dtHLAGM 

dtInputlSense 

dtInputPLIB 

dtInspectorQt 

dtLMS 

dtNet 

dtNetGM 

dtQt 

dtSeript 

dtTerrain 

dtUtil 

NA 

psGeodeTransform 

sigslot 


31 

98 

97 
55 
14 

98 
16 
19 
46 

1 

1 

21 

13 

3 

11 

46 

1 

54 

96 

57 

1 

43 


Kernel 


369 


dtCore 

dtDAL 

dtGame 


155 

111 

103 
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Table 18: Detailed listing of reuse ratings for the DMZ framework 


Code Classes WMC DIT NOC CBO RFC 



475 


DMZ 


Middleware 


319 


frameworks/arehive 

frameworks/audio 

frameworks/entity 

frameworks/event 

frameworks/input 

frameworks/net 

frameworks/obj ect 

frameworks/qt 

frameworks/render 

frameworks/weapon 


11 

11 

28 

10 

18 

71 

27 

80 

57 

6 


Kernel 


156 


foundation/libs 

foundation/plugins 

kemel/runtime 

kemel/system 

kemel/types 


26 

10 

70 

19 

31 


Table 19: Summary listing of reuse ratings for Delta3D and DMZ frameworks 


Code 

Classes 

WMC 

DIT 

NOC 

CBO 

RFC 

Delta3D 

1,191 

I 

II 

I 

II 

II 

Middleware 

822 

I 

II 

I 

II 

II 

Kernel 

369 

I 

II 

I 

II 

II 

DMZ 

475 

I 

I 

I 

II 

II 

Middleware 

319 

I 

I 

I 

II 

II 

Kernel 

156 

I 

I 

II 

I 

I 
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4. 


Results 


The results of the reuse ratings and the process of interpreting and studying the 
results provides valuable insight into the software being assessed. This helps the acqui¬ 
sition professional better understand the frameworks under consideration to make a more 
informed decision. 

To aid in the comparison between DeltaSD and DMZ, we normalize the DMZ num¬ 
ber of classes according to DeltaSD’s size. To adjust for the fewer classes in DMZ (about 
60% fewer), we solve for x in the relationship given by Equation 8 where M is the number 
of DMZ classes in a given subset, and Ni is the number of classes in each framework with 
i G {DMZ, DeltaSD}. 


M X 

N-= N- 

I^DMZ I^DeltaSD 

For example, if we counted 24 classes in DMZ and 70 classes in DeltaSD for some 
grouping, then to find the adjusted DMZ value x that can be compared to 70, we take the 
number of DMZ classes in the grouping M = 50, the total number of classes in DeltaSD 
^DeltaSD = 1191, thc total number of classes in DMZ Ndmz = 475, and compute x as 
shown in Equation 9: 


M 


N 


DMZ 

l^DeltaSD 

X = 

l^DeltaSD 


24 

X = 

9 _ 

475 

X = 

60.2 


M 


N 


DMZ 


(9) 


We can then compare the adjusted DMZ value 60.2 to the DeltaSD value 70. When 
this adjustment is made, it will be noted in the table or figure that accompanies it. 
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Assigning notional weights to the weighting funetion allows for further analysis. 
These weights would be eustomized aeeording to the needs of the program manager. As¬ 
sume a weight of 1 for 1,2 for II, and 3 for III (Equation 10). 


^lTa(^lTa) ^ 


1 , 

2 , 

3 , 


if = I 

if = II , V 1, m 

if Rim = III 


( 10 ) 


With a set of possible weights provided and foeusing on the development opera¬ 
tions, we ean plot how the model differentiates between frameworks. Figure 10 shows the 
results of plotting the weighted values as a staeked bar ehart. 


Comparison of Weighted Ratings for Reuse 



WMC 

DIT 

NOC 

CBO 

RFC 


Delta3D DMZ Delta3D DMZ Delta3D DMZ 
Overall Middleware Kernel 


Figure 10: The two frameworks differ with respeet to reuse. 


a. Weighted Methods per Class 

Both frameworks are rated Iwmc for WMC, whieh eounts the number of 
methods in eaeh elass, revealing that in general both frameworks have reasonably-sized 
elasses that are not too eomplex. Plotting the number of elasses against their WMC values 
on a logarithmie seale (Figure 11) reveals that DMZ has a slightly higher average WMC 
than DeltaSD. It is interesting to take a look at the top ten “hot spot” elasses with respeet to 
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WMC (Table 20). The first two classes dtCore: : RefPtr and dtCore : :ObserverPtr 
inherit from Open Scene Graph classes and perhaps could be excused. Despite the fact that 
DMZ has a higher average WMC, only two DMZ classes make it into the top ten (the next 
one is ranked #18). This may be another result of the component approach to development 
which values many, same-sized, composable components over “kitchen-sink” objects. 


Distribution of Ciasses by WMC 



Figure 11: Number of classes per framework plotted by WMC reveals that DMZ has a 
higher average WMC. 


b. Depth of Inheritance Tree 

DeltaSD is rated IIdu , and DMZ is rated Id it for DIT. It is expected 
that component based architectures are “flatter” with respect to inheritance (Qingqing & 
Xinke, 2009), and the findings here are consistent with that observation. Lower DIT values 
indicate that classes tend to be near the top of the inheritance chain and that inheritance is 
not the primary means by which functionality is extended. Recall that lower DIT values 
suggest lower dependencies on other classes and thus more loosely coupled code. 
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Table 20: The top ten classes with the highest WMC values reveals more potentially- 
problematic classes in DeltaSD. 


Rank 

Class 

Framework 

Layer 

WMC 

1 

dtCore::RefPtr 

Delta3D 

Kernel 

763 

2 

dtCore: :ObserverPtr 

Delta3D 

Kernel 

186 

3 

dtGame:: GameManager 

Delta3D 

Kernel 

135 

4 

dtGame:: DeadReckoningHelper 

Delta3D 

Kernel 

121 

5 

dmz: :ObjectModuleBasic 

DMZ 

Middleware 

114 

6 

dtUtil: :DataStream 

Delta3D 

Middleware 

102 

7 

dtHLAGM: :HLAComponent 

Delta3D 

Middleware 

87 

8 

dtUtil::KDTree 

Delta3D 

Middleware 

84 

9 

dmz:: Obj ectModule 

DMZ 

Middleware 

78 

10 

dtAnim: :Cal3DModelWrapper 

Delta3D 

Middleware 

77 


Figure 12 reveals that DeltaSD has a greater number of classes with high 
DIT values than DMZ. Table 21 lists the top ten classes ranked by DIT values as well as 
the highest-ranked DMZ class, which arrives after the top 380 classes with DIT values 
[3..8]. Between the two frameworks there are 289 classes with DIT = 2. 

c. Number of Children 

Both frameworks are rated Inoc for NOC indicating good levels of inher¬ 
itance. Recall from Section 1 that some inheritance suggests good reuse but that greater 
values of NOC may suggest improper abstraction of classes (S. R. Chidamber & Kemerer, 
1994). 

Plotting the number of classes against their NOC values on a logarithmic 
scale (Figure 13) reveals that Delta3D and DMZ are fairly matched in NOC with 84.0% 
of the Delta3D classes and 82.7% of the DMZ classes at NOC = 0. A look at the top 
ten classes ranked by NOC (Table 22) reveals a spike in the dmz : : plugin class with a 
whopping 167 children—almost double the number two class. The high NOC count for 
dmz: :plugin reminds us that in a component architecture we expect to see all of the 
components derive from the base component (plugin) class, not from other components 
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Distribution of Ciasses by DIT 



Figure 12: Number of classes per framework plotted by DIT reveals that DeltaSD has a 
greater number of classes with high DIT values. 


Table 21: The top ten classes with the highest DIT values reveals more deep-inheritance 
classes in DeltaSD. 


Rank 

Class 

Eramework 

Eayer 

DIT 

1 

dtActors:: WaterGridActor 

Delta3D 

Middleware 

8 

2 

dt Ac tors:: WeatherEnvironment Actor 

Delta3D 

Middleware 

8 

3 

dtActors: :TaskActorGameE vent 

Delta3D 

Middleware 

8 

4 

dtActors:: SkyDomeEnvironment Actor 

Delta3D 

Middleware 

8 

5 

dtActors: :TaskActorOrdered 

Delta3D 

Middleware 

8 

6 

dtActors: :TaskActorRollup 

Delta3D 

Middleware 

8 

7 

dtAnim: :Cal3DGame Actor 

Delta3D 

Middleware 

7 

8 

dtActors:: Director Actor 

Delta3D 

Middleware 

7 

9 

dtActors: :WaterGridActorProxy 

Delta3D 

Middleware 

7 

10 

dtActors: :DistanceSensorActor 

Delta3D 

Middleware 

7 

380 

sigslot::signal8 

Delta3D 

Middleware 

3 

381 

dmz: :RenderPluginEventOSG 

DMZ 

Middleware 

2 
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that have been extended. This is the proper behavior for a eomponent based arehiteeture, 
and the metries eonfirm that DMZ matehes this behavior. 


Distribution of Ciasses by NOC 



Figure 13: Number of elasses per framework plotted by NOC reveals that DeltaSD and 
DMZ are fairly matehed in NOC values. 


d. Coupling between Objects 

Both frameworks are rated IIcbo for CBO. In Table 19 we observe that 
DeltaSD seores IIcbo for both Kernel and Middleware, while DMZ seores different be¬ 
tween Kernel and Middleware, IcBO and IIcbo > respeetively. This might not surprise us 
sinee we know that the DMZ Kernel is fundamentally different than its Middleware. Reeall 
that a higher CBO value suggests greater eomplexity, greater interdependenee, and greater 
fragility when reusing eode. 

Plotting the number of elasses against their CBO values on a logarithmie 
seale (Figure 14) reveals that DeltaSD has a greater number of low-CBO elasses and that 
CBO values are fairly evenly distributed aeross DMZ. However, it is interesting to take a 
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Table 22: The top ten classes with the highest NOC values reveals a close matching between 
DeltaSD and DMZ. 


Rank 

Class 

Framework 

Layer 

NOC 

1 

dmz:: Plugin 

DMZ 

Kernel 

167 

2 

dtUtil: :Enumeration 

Delta3D 

Middleware 

86 

3 

dmz:: ObjectObserverUtil 

DMZ 

Middleware 

71 

4 

dmz:: Times lice 

DMZ 

Kernel 

60 

5 

dtUtil: :Exception 

Delta3D 

Middleware 

57 

6 

dmz: :MessageObserver 

DMZ 

Kernel 

37 

7 

dmz: :InputObserverUtil 

DMZ 

Middleware 

36 

8 

dtCore::Base 

Delta3D 

Kernel 

34 

9 

dtCore:: Transformable 

Delta3D 

Kernel 

25 

10 

dtDirector:: ActionNode 

Delta3D 

Middleware 

23 


look at the top ten classes ranked by CBO. Table 23 reveals that DeltaSD also has some of 
the highest-CBO values. The next DMZ class is ranked #16. 

e. Response for a Class 

Both frameworks are rated IIrpc for RFC, and again we observe that the 
DMZ Kernel is rated slightly different at Irfc • 

Plotting the number of classes against their RFC values on a logarithmic 
scale (Figure 15) reveals that DeltaSD has a number of very small, almost empty, classes 
and that DMZ otherwise has lower RFC values. Table 24 lists the top eleven classes ranked 
by RFC. The eleventh is added to the table because it is the first appearance of a DMZ 
class. Recall that a higher RFC value suggests greater complexity, greater potential for 
unintended consequences, and greater fragility when reusing code. 

/, Top 100 Classes 

Looking at the top 100 classes sorted in turn by each metric in both frame¬ 
works shows that DMZ has a smaller presence in these potential hot spot classes than 
DeltaSD (Table 25 and Figure 16). 
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Distribution of Ciasses by CBO 



Coupling Between Objects (CBO) 

Figure 14: Number of classes per framework plotted by CBO reveals that DeltaSD has a 
greater number of low-CBO classes than DMZ. 


Table 23: The top ten classes with the highest CBO values reveals that DeltaSD has more 
of the highest-CBO values than DMZ. 


Rank 

Class 

Framework 

Layer 

CBO 

1 

dtHLAGM:: HL AComponent 

Delta3D 

Middleware 

69 

2 

dtGame:: GameManager 

Delta3D 

Kernel 

63 

3 

dtDAL:: ActorPropertySerializer 

Delta3D 

Kernel 

61 

4 

dtDAL: :NamedGroupParameter 

Delta3D 

Kernel 

49 

5 

dtDirector:: DirectorEditor 

Delta3D 

Middleware 

48 

6 

dmz: :ObjectModuleBasic 

DMZ 

Middleware 

46 

7 

dtDirector:: DirectorEditor 

Delta3D 

Middleware 

46 

8 

dt Actors:: WaterGridActor 

Delta3D 

Middleware 

46 

9 

dtCore: :StatsHandler 

Delta3D 

Kernel 

45 

10 

dtGUI::GUI 

Delta3D 

Middleware 

44 
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Distribution of Ciasses by RFC 



Response for a Class (RFC) 

Figure 15: Number of classes per framework plotted by RFC reveals DeltaSD with some 
small classes but otherwise beat by DMZ. 


Table 24: The top eleven classes with the highest RFC values reveals that DeltaSD has 
some of the highest-RFC classes. 


Rank 

Class 

Eramework 

Eayer 

REC 

1 

dtCore::RefPtr 

DeltaSD 

Kernel 

852 

2 

dt Actors: :WeatherEnvironment Actor 

Delta3D 

Middleware 

224 

3 

dt Actors:: WaterGridActor 

Delta3D 

Middleware 

219 

4 

dt Actors:: TaskActorGameEvent 

Delta3D 

Middleware 

217 

5 

dt Actors:: TaskActorOrdered 

Delta3D 

Middleware 

210 

6 

dt Actors:: TaskActorRollup 

Delta3D 

Middleware 

207 

7 

dt Actors:: TaskActor 

Delta3D 

Middleware 

205 

8 

dt Actors:: SkyDomeEnvironmentActor 

Delta3D 

Middleware 

204 

9 

dtActors: :CoordinateConfigActor 

Delta3D 

Middleware 

204 

10 

dt Actors:: Director Actor 

Delta3D 

Middleware 

203 

11 

dmz: :ObjectModuleBasic 

DMZ 

Middleware 

202 
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Figure 16: DMZ has fewer classes in the top 100 (except RFC), sorted descending by 
metric, than DeltaSD, even adjusting for DMZ being 60% smaller than DeltaSD. 


Table 25: DMZ has fewer classes in the top 100 (except RFC), sorted descending by metric, 
than DeltaSD, even adjusting for DMZ being 60% smaller than DeltaSD. 


Framework 

WMC 

DIT 

NOC 

CBO 

RFC 

DeltaSD 

55.8 

100.0 

57.2 

61.5 

44.7 

DMZ 

44.2 

0.0 

42.8 

38.5 

55.3 
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C. SUMMARY 


We have successfully shown that our reuse model differentiates between two visual 
simulation frameworks and that we gain valuable insight in the process of applying and 
interpreting the model. The model contributes a new approach and tool for program man¬ 
agers and others to assess the nature of visual simulation frameworks with respect to the 
potential for reuse. 

The model does not present as dramatic a difference between the frameworks as 
was expected. The proponents of component architecture tout the focus on decoupling, but 
the model seemed to focus more on the reusability of the underlying objects on which the 
components were built than the components themselves. For that, new metrics of agility 
are developed in Chapter V. 
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V. AGILITY 


In this chapter we develop and apply a model for assessing agility based on the 
definition and need established in Chapter II: Literature Review. The model uses new 
metrics that were created out of necessity for lack of any established metrics. We conduct 
a study in which we show that the model differentiates between two visual simulation 
frameworks and dramatically illustrates the differences in the two architectures. 

A. DEVELOPING THE AGILITY MODEL 

We learned that “‘agility is a very seductive word” and that there are many “personal 
definitions” of it (Dove, 1994), but we require a model that is more precise. Measuring 
agility as we define it, by code being easily reconfigured, repurposed, or integrated, is not as 
straightforward as the static analysis we used for openness or reuse. We have defined agility 
in terms of actions—^reconfigure, repurpose, integrate—and so agility must be measured “in 
flight” as actions take place. 

1. Measuring Agility 

Agility is sometimes linked to component based programming, and there has been 
some effort in developing complexity and other metrics specifically for components (Sharma 
& Kumar, 2007; Ismail, Wan-Kadir, Saman, & Mohd-Hashim, 2008; Qingqing & Xinke, 
2009). These metrics try to do for components what CK metrics do for objects and classes, 
but they do not address our specific question of agility. 

We could find no literature offering metrics for measuring software agility, but 
Lanman and Proctor (2009) remind us of the value of swapping out components in a sys¬ 
tem. Therefore, swapping out functionality in software is where we will turn. This action 
touches on reconfiguring, repurposing, and integration. 

Being able to swap out components is a useful thing. It is easy for developers to 
think that once a piece of code is working, they will never go back to it, although this could 
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be due to nothing more than bum out from working with the same code for a long time. 
Whatever the reason code inevitably needs to be replaced or upgraded, because require¬ 
ments inevitably change. Developers should be planning on change as the only constant, 
and something that improves the change process is worth considering. 

It is not feasible to measure directly the effort required to swap out a portion of a 
simulation framework. Effort could be measured in time or money. It would be wasteful 
of both to go through the whole process of swapping out code just to see how hard it was. 
Instead we require metrics that will estimate the effort that would be required to execute 
the swap. 

a. Included Files 

We can estimate the effort of swapping out a piece of code by counting the 
connections to those files that are being replaced. Many programming languages have an 
include or import statement that makes available the functionality of an external piece 
of code. The more code we have that connects to this swapped out code, the greater the 
effort required to complete the swap. We therefore propose four metrics that measure the 
extent to which files are included that will need to be removed as part of the swapping 
process (Table 26). 

Functionality F is being swapped out. L is the set of all lines of code in the 
project. C is the set of all classes in the project. Ip is the set of all include statements 
for functionality F. Li^ is the set of all lines of code that include functionality F. Cp is 
the set of all classes that include functionality F. |X| is the cardinality of set X. 

Table 26: Agility metrics related to the files that are included. 


Metric 

Formulation 

Lines that Include F (LI) 

LI = iLi^| 

Percent of Lines that Include F (PLI) 

PLI = |Li,|-|L| 

Classes that Include F (Cl) 

CI = |Ci,| 

Percent of Classes that Include F (PCI) 

PCI = |Ci,|-|C| 
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It is possible that some files may have # include statements but do not 
actually use functionality F in the file. While such files should be easier to fix than others, 
they still must be inspected and altered for the removal of functionality F to be complete, 
and so the metrics retain their usefulness even in this situation. 

b. References 

Counting the # include statements is a good start and provides a good, 
coarse metric for estimating the effort required for swapping out functionality, but there 
are at least two issues that it leaves unaddressed: the case of unnecessary, “left over” 
# include statements abandoned in the code and the amount of code within each file that 
actually uses the functionality. To address these, additional metrics that count the actual 
references to elements, e.g., functions or variables, from functionality F are added (Table 
27). 

Rp is the set of all operations that use functionality F. Lr^ is the set of all 
lines of code that use one ore more operations in Rp. Cr^ is the set of all classes that use 
one ore more operations in Rp. 

Table 27: Agility metrics related to references made. 


Metric Formulation 

Lines with References to F (LR) LR = ILr^I 

Percent of Lines with References to F (PLR) PLR = ILr^I |L| 
Classes with References to F (CR) CR = ICr^I 

Percent of Classes with References to F (PCR) PCR = ICrpI |C 


2. Model for Assessing Agility 

To assess agility we select a functionality and estimate the effort to swap it out. The 
eight agility metrics are calculated, but the actual process of swapping out the functionality 
is not completed as that would entail unnecessary cost. 
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Unlike the openness and reuse models we have no information on which to create 
criteria against which to measure these metrics objectively. Instead multiple frameworks 
can be compared by comparing the metrics (Table 28). 

Table 28: Agility metrics can be compared side by side. 



Framework 

Metric 

Framework 1 

Framework 2 

Lines of Code, L 

xx,xxx 

xx,xxx 

Number of Classes, C 




Lines with Includes, LI 

Percent of Lines with Includes, PLI 
Classes with Includes, Cl 

Percent of Classes with Includes, PCI 

Lines with References, LR 

Percent of Lines with References, PLR 
Classes with References, CR 

Percent of Classes with References, PCR 


B. STUDY 3: ASSESSING AGILITY 

To demonstrate the feasibility of this assessment model, two simulation frameworks 
DeltaSD and DMZ (described in Sections 2 and 3) are analyzed with the agility model by 
estimating the effort required to swap out the rendering engines. The rendering engine is 
perhaps the most complex portion of a visual simulation framework, so performing well 
in this test is a significant challenge. This is a worst case scenario. In both frameworks, 
the rendering engine is OSG. This helps provide a clean environment for verifying the 
usefulness of the model. 

1. Methodology 

To calculate the metrics in Tables 26 and 27, scripts (Appendix B) were run against 
the source code in each framework. 
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After the counting metrics are calculated, the percentages are calculated based on 
the number of lines of code or number of classes, as appropriate, and as given by the tool 
Understand used earlier in this research. 

The data was then compiled in spreadsheets to be aggregated by framework, layer, 
or other division. Calculations, charts, and analysis was conducted from within these 
spreadsheets. 

2. Results 

The experiment is quick to execute, although the effort represented by the metrics 
is still considerable. A more thorough experiment, though of questionable additional value, 
would be to fund the teams of developers to implement this change and record the time and 
money that was required. Table 29 shows the results of the calculations. 

Table 29: The agility model suggests almost an order of magnitude improvement of DMZ 
over DeltaSD. 



Framework 

Metric 

DeltaSD 

DMZ 

Lines of Code, L 

160,705 

98,336 

Number of Classes, C 

1,191 

475 

Lines with Includes, LI 

935 

158 

Percent of Lines with Includes, PLI 

0.6 

0.2 

Classes with Includes, Cl 

310 

31 

Percent of Classes with Includes, PCI 

26.0 

6.5 

Lines with References, LR 

4,687 

680 

Percent of Lines with References, PLR 

2.9 

0.7 

Classes with References, CR 

259 

31 

Percent of Classes with References, PCR 

21.7 

6.5 


a. Dramatic Differences 

The results are also presented graphically in Figure 17. A striking differen¬ 
tiation can be seen in all four charts. The upper left chart, for example, reads, “Regard- 
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ing includes, DeltaSD had 935 OSG includes, and DMZ had 158. Regarding references, 
DeltaSD had 4,687 references, and DMZ had 680.” This is almost an order of magnitude 
difference. It represents actual time programmers would need to spend pulling out Open 
Scene Graph and putting in something else. Removing size from the equation, the lower 
right chart is also compelling. It reads, “Regarding includes, 26.0% of DeltaSD classes had 
OSG includes, and 6% of DMZ classes had OSG includes. Regarding references, 22% of 
DeltaSD classes had OSG references, and 6% of DMZ classes had OSG references.” This 
is a significant difference. 


Count 


Percent 



Includes References 


Includes References 


■ Delta3D 


DMZ 


lower is more agile 


Figure 17: Whether by lines of code, by class, by raw count, or by percentage, the compo¬ 
nent based DMZ reveals itself to be more agile than DeltaSD. 


Yet another way to look at the results is with a Kiviat diagram (Figure 18), 
normalized with the DeltaSD scores around the perimeter. This again shows a dramatic 
difference between the two frameworks when measuring agility. 
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Lines with #includes (LI) 



Percent Lines with #includes 
, (PLI) 


Percent Classes with #includes 


Classes with #includes (Cl) 


(PCI) 


Lines with OSG References (LR) 


Figure 18: The estimated effort to swap out the rendering engine of DMZ (inner polygon, 
as a pereent of DeltaSD) is eonsiderably less than DeltaSD. 

b. Includes vs. References 

We might expeet to see some eorrelation between some of the “files in- 
eluded” and “referenees” metries, and that seems to be borne out. For example, DMZ has 
31 elasses that have OSG include statements and 31 elasses that make OSG referenees, 
but breaking out the metries this way also reveals some ineonsisteneies in Delta3D. There 
are 310 elasses that have OSG include statements but only 259 elasses with OSG ref¬ 
erenees. This suggests “left over” eode in Delta3D that eould be eleaned up. A future 
study with more frameworks might determine if the include and reference metries eould be 
eonsolidated. 
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c. Absolutes vs. Percents 


The metrics which provide absolute counts and the metrics with percentages 
offer different insight and should both be retained. The absolute counts, e.g., LR and CR, 
are a doorway to estimating the time and money that would be required to make the changes 
represented in the study. The percentages provide a convenient, normalized metric that is 
more appropriate for assessing the quality of the code independent of its size. 

C. SUMMARY 

We have successfully shown that our agility model differentiates between two vi¬ 
sual simulation frameworks and that we gain valuable insight in the process of applying 
and interpreting the model. The model contributes a new approach and tool for program 
managers and others to assess the nature of visual simulation frameworks with respect to 
openness. 
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VI. CONCLUSION 


A. REVIEW 

Although there is much guidance in the literature and in government documents for 
acquisition professionals regarding openness, reuse, and agility, there are no quantitative 
models that a PM could use to compare visual simulation architectures. This research 
contributed three models that provide quantitative differentiation for openness, reuse, and 
agility. 

For each objective, a different amount of help was available from the literature 
(Table 30). With openness, the literature had taxonomies and terms, but this research de¬ 
veloped the metrics and rolled them into a useful model. With reuse, the literature had 
definitions and even metrics, but this research brought the metrics together into a coherent 
model. With agility, the literature provided very little foundation, except to establish that 
people wanted “agility.” This research provided a working definition, developed the met¬ 
rics, and made some sense of the metrics all from scratch. For each objective this research 
included a study to demonstrate feasibility, and those were successful. 

Table 30: For each objective, there was a different amount of help available from the liter¬ 
ature. 



Openness 

Reuse 

Agility 

Define terms 

Lit. 

Lit. 

• 

Develop metrics 

• 

Lit. 

• 

Build quantitative model 

• 

• 

• 

Perform feasibility study 

• 

• 

• 
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B. DISCUSSION 


1. Choice of Models 

Prior to this research, there were few methodologies available to quantitatively iden¬ 
tify the advantages of one architecture over another. The comparisons of one architecture 
to another seemed to hinge more on “brochure-ware” than scientific analysis. There did 
not seem to be a rigorous testing method for assessing attributes of simulation software 
that people supposed were important. 

Openness and reuse were identified as candidate attributes. There is a lot of de¬ 
mand in the government for these qualities. However, these qualities have not been rig¬ 
orously defined. There was much literature addressing various meanings of openness and 
various attributes related to reuse. There were definitions for the terms and some attempts 
at quantifying them, but there was nothing available to assess visual simulation software 
architectures. 

The openness model developed in this research differentiated between the two 
frameworks studied. Both frameworks had similar open source licenses. Therefore, the 
strongest differentiation was made regarding standards and innovation. The model revealed 
significant differences in the consistency across the layers and the development operations. 
These differences would not be revealed by a licensing-only approach to openness. 

The reuse model was the most surprising. Component based architectures stress 
decoupling. This suggested that the CK metrics that relate to coupling would reveal this 
strength of components over objects. This was not so. The component architecture was not 
a clear winner. Although the two frameworks were significantly different in their architec¬ 
ture, the reuse model showed only marginal differences. 

It is the composability of component architectures that is attractive. Although both 
objects and components are meant to be reusable and both encourage programming to clear 
interfaces, components go a step further and abstract even the communication mechanisms. 
Components try to reduce coupling not by eliminating coupling itself but by changing 
the kind of coupling that happens. Objects can be though of as puzzle pieces—clearly 
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defined, well-documented, interchangeable even—but puzzle pieces nonetheless. Consider 
components to be Tinkertoys with a variety of shapes but very few ways to connect them 
(Figure 19). 



Object Oriented 



Component Based 


Figure 19: Objects can be likened to puzzle pieces that fit neatly together, and components 
can be likened to Tinkertoys with their various shapes connected by simple means. 


This realization moved the research in a new direction toward agility. Measuring 
agility required measurements to be taken “in motion” rather than through static code anal¬ 
ysis. This produced the methodology of swapping out a portion of a framework’s function¬ 
ality and estimating the effort of this action. The difference between the two frameworks 
studied was significant. 

2. Additional Agility Metric 

Some alternate analysis for assessing agility was introduced too late in the research 
for in depth study, but a brief summary is presented here. The central question revolves 
around how much variation is seen in the OSG references that are made within the frame¬ 
works. The idea is that a developer would have an easier time swapping out OSG from a 
class that had 100 references to the same OSG class, than a class with 1 reference each to 
100 different OSG classes. 

To investigate whether or not this could differentiate the two frameworks, the agility 
study was augmented to include a new metric. Response Entropy for a Class (REG). This 
metric is similar to the REG CK metric, in that it represents a kind of “damage control” 
estimate. It is also similar in measurement technique to CBO. To calculate REG, for each 
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class in the framework that makes references to the functionality being swapped out, count 
the number of different classes from that functionality that are referenced. For example, if 
the OSG class osg: : Vec3 was referenced 20 times within a class, and that was the only 
class referenced from OSG, then REG for that class would be 1. The RFC values were 
calculated for DeltaSD and DMZ and aggregated by layer. 

Table 31 shows the REG values for the classes that had OSG references. For com¬ 
parison, the GBO values are also presented both for the classes that had OSG references as 
well as for all the classes in the framework. Also of interest is the observation that the GBO 
average values for the classes with OSG references was much lower than the averages for 
the entire frameworks. 

Table 31: Agility REG values compared to GBO values for Delta3D and DMZ 





GBO 


Gode 

Glasses 

REG 

These Glasses 

All Glasses 

Delta3D 

259 

8.2 

15.3 

7.43 

Middleware 

167 

6.0 

15.5 

7.32 

Kernel 

92 

12.3 

15.1 

7.69 

DMZ 

31 

10.8 

17.6 

8.37 

Middleware 

31 

10.8 

17.6 

10.24 

Kernel 

- 

- 

- 

4.54 


The REG appears to provide differentiation. It also provides an interesting way to 
reason about the effort required for swapping out parts. As such this metric deserves further 
exploration in future work. 

The correlation between REG and GBO (Figure 20) is expected. Although REG is 
related to REG for intent, it is also related to GBO in actual measurement. An increase in 
the REG necessarily implies an increase in GBO. 
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Response Entropy for a Class (REC) 


■DeltaSD 

DMZ 


Figure 20: As expected, REC correlates to CBO 


3. Uniformity of Design 

An unexpected discovery during the openness study (Section B) was that the com¬ 
ponent framework DMZ had more consistent ratings across software layers and develop¬ 
ment operations than the conventional object based DeltaSD. Recall that with regard to 
standards DeltaSD scored Ig , IIj , and Ills for integrating, extending, and modifying 
the middleware, while DMZ scored a consistent Ig for all three because all interactions 
with DMZ revolve around configuring its plugins and manipulating data in its Object Mod¬ 
ule. 

The value of this consistency is hard to determine. Because the integrate operation 
is the primary means by which a programmer uses a framework, this consistency may 
not be important to some developers. However, that view might be short-sighted. Today 
a framework may do just what the programmer needs it to do, but tomorrow it may fall 
short. An architecture with a consistent means for performing all development operations 
may help protect against the lock-in often seen today. Simply being able to perform all 
development operations is a win, but consistency therein is even better. 
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Consider the case of Virtual Battlespace 2 (VBS2), the “video game” from Bohemia 
Interactive developed for the military. VBS2 has an Application Scripting Interface (ASI) 
that allows developers to add features and behaviors. Evertsz and Ritter (2009) combine the 
cognitive architecture CoJACK with VBS2 to create suicide bomber scenario where fear 
and morale are modeled in the virtual actors. They remark that they are limited by what data 
and functionality is available in the scripting language. Not all elements of the simulation 
are accessible. Bohemia Interactive released VBS2Fusion which is advertised as protecting 
developers from having to learn their scripting language by letting them use C++ instead. 
It has expanded support for accessing data within the simulation (SimCentric, 2011). The 
impact of this new product, purchased separately from VBS2, has yet to be determined, but 
military customers are clearly held over a barrel as they try to integrate, extend, and modify 
their simulations. 

The advantage of a uniform design may go deeper still. Extended applications (third 
party simulations) built with DMZ are subject to the same uniformity as the middleware. 
This means that, barring poor licensing negotiations with vendors, simulations built with 
an architecture like DMZ allow the government not only greater latitude in expanding the 
power of the framework but also greater ability to pick out and repurpose components from 
third party simulations. This may take us one step closer to avoiding the dreaded, “But we 
already paid for that!” complaint. 

4. Helping the Three Program Managers 

Chapter I introduced three PMs with three different needs for visual simulation as 
examples of stakeholders who could benefit from this research. One PM had a project with 
a horizon of 15 years or more. Another expected many other projects to fork off from the 
original. The third needed a quick and dirty simulation to answer a narrow and specific 
question. These three PMs might not weight the metrics the same and might eliminate 
certain elements of the model. 
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These three PMs were given weights for the openness and reuse models based on 
the things that were important to them. The models differentiated between the two frame¬ 
works, but with the weights applied, the model was able to present results from eaeh PM’s 
perspeetive. 

Figure 21 shows that beeause of the way openness matters to the first two PMs, 
DMZ seores higher than DeltaSD. For the PM who has an exereise eoming up and just 
wants a quiek analysis, openness is not as important, and the differenees between the frame¬ 
works are marginalized. 

With reuse the two frameworks were mueh more similar. Two things ean be ob¬ 
served in the ehart on the right. One is that two of the PMs eare more about reuse than the 
third. This is similar to openness. Also the differenees between the two frameworks are not 
great, sinee the differenee in the ratings were not so dramatie with reuse. 


Framework Scores Based on Openness 


Framework Scores Based on Reuse 
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Figure 21: Different priorities among program managers may eause them to value open¬ 
ness, reuse, and agility—and henee different arehiteetures—differently. 


Agility is the least mature model, for obvious reasons. It has the least support 
from literature. The Kiviat diagram in Figure 18 is a eompelling visual for PMs who are 
eoneemed about agility. 
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C. CONCLUSIONS 


Openness, reuse, and agility are indeed important attributes of visual simulation 
frameworks, and they can be used to assess simulation frameworks and their underlying 
architectures. The three models developed in this research were successful at differen¬ 
tiating the two visual simulation frameworks studied and provided insight that could help 
acquisition professionals make decisions regarding their choice of simulation architectures. 

In the literature we see that openness, reuse, and agility are important and that 
there are no suitable models for assessing visual simulation with respect to these qualities. 
Breaking openness into standards, licensing, and innovation on one axis (Maxwell, 2006) 
and software layers and development operations on two more axes (Anvaari & Jansen, 
2010), provides a powerful space in which to characterize visual simulation, and possibly 
other, software. The venerable CK metrics (S. Chidamber & Kemerer, 1991) are well- 
tested and have many years of use to their credit. There is little to help build an agility 
model. 

The three-axis openness model was a more powerful differentiator than was ex¬ 
pected. It distinguished between two open source frameworks in the areas of open standards 
and open innovation. It revealed interesting features about one component based architec¬ 
ture regarding its uniformity of design (Section 3) that may help reduce the painful effects 
of legacy code in the future. Weighting the ratings allows for customizing an analysis of 
frameworks based on the particular needs of the project at hand. 

The reuse model was the weakest differentiator of the three but did succeed in high¬ 
lighting potential hot spots in the frameworks. In one case examining average metric values 
made one framework more appealing while examining the worst ranking classes made the 
other framework more appealing (Section a). The metrics related to coupling did not offer a 
convincing case for the advantages of components of objects, as was supposed. This effect 
encouraged the development of a third model based on agility (Section 1). Weighting the 
ratings allows for customizing an analysis of frameworks based on the particular needs of 
the project at hand. 
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The agility model was built without the aid of supporting literature but still revealed 
the most dramatic differences between the two frameworks studied. The estimated differ¬ 
ence in effort between the two frameworks was almost an order of magnitude. Some of the 
advantages that proponents of component architectures like to promote regarding compos- 
ability and flexibility were borne out in this study (Section 2). 

D. FUTURE WORK 

As successful as the three models of openness, reuse, and agility are, there are other 
models for these same important areas that could be developed and compared to the three 
in this dissertation. This is the first attempt to provide quantitative models for assessing 
visual simulation frameworks, and there may be room for improvement. 

The metrics in the reuse model could be replaced with other software metrics linked 
to reusability. The CK metrics were selected because they have stood for a long time, are 
respected, and have been validated in the literature, but there are dozens of other metrics 
(Xenos et ah, 2000; Kitchenham, 2010), largely untested except for the papers wherein 
they were introduced, and some of them may have some advantages. Metrics that may 
differentiate the models better along reuse should replace the metrics used here. 

The strong differences in agility between the two frameworks suggest that there is 
potential to explore this further and develop more advanced and detailed models that break 
up agility into finer grains as we have done with the openness model. The characteristics 
that Qumer and Henderson-Sellers (2006b) uses—flexibility, speed, leanness, learning, and 
responsiveness—in measuring agility in companies may provide an alternative path for 
defining agility in software as well. 

Other areas besides openness, reuse, and agility could be pursued. Maintainability 
and security are two very different characteristics at which people nod their head saying 
yes, they want more of it, and these are but two of the many desirable qualities of software 
that could benefit from clear definitions and assessment tools. 
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The final goal of this dissertation and related future work is to provide PMs with 
quantitative tools that they can use to aid in their decision making about visual simulation 
development. Rigorous models that lead to better architectures will help the DoD build or 
buy better software. 
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A. APPENDIX 


A. SCRIPTS FOR CALCULATING REUSE METRICS 
1. Parent Script 

The following script fires off all the required scripts for generating metrics. 

#1/hin/sh 

# RunAll. sh 

if [ — It 1 ]; then 

echo ’’USAGE: ^$ (basename ^$0) ^ under stand —database . udb” 

echo ’’Be^sure^also^that ^you ^have ^generated ^reports ^in ^Understand:” 

echo ’’^^Reports ^menu^—>^Generate ^Reports” 

echo ’’Probably^best^to ^run ^from ^the^directory^with^the^. udb ^ f i 1 e . ” 

exit 

fi 

UDB=”$1” 

UDBDIR=$ ( dirname ”$UDB”) 

UDBBASE=$ ( basename —s .udb ”$UDB”) 

# Convert the Understand Reports file to several CSV files 
if [ -f ”${UDBDIR}/${UDBBASE}. txt” ]; then 

' / metrics / scripts / Convert—Understand—Report—To—CSV. sh ”${UDBDIR}/${UDBBASE} . txt” 

echo 

else 

echo ”$(tput^setaf^l)Missing^report^file .$(tput^sgr0)” 
echo ”^^You^need^to^generate^reports^in^Understand:” 
echo ’’^^Reports ^menu^—>^Generate ^Reports” 

echo ” ^^The^ file ^ should ^be^named^S (tpu t ^ se taf ^2)${UDBDIR}/${UDBBASE} . txt$ (tput^sgrO)” 
exit 1 
fi 


# Runs some custom metrics using the Understand Perl API 
' / metrics / scripts /RunAllPerl . sh ”$UDB” 


# Merge the CSV files that are keyed off of classnames 

echo ”Merging^csv^files^to^$(tput^ se t af ^3) ${UDBDIR}/${UDBBASE}—Merged_Class_Metrics .csv$(tput^sgrO)” 
'/metrics / scripts/MergeCsv . sh \ 

${UDBDIR}/${UDBBASE}—Class {_00,} _Metrics_Report . csv \ 

${UDBDIR}/${UDBBASE}—{cj _classmetrics , coupling , c_class_filenames , c_halstead }. csv \ 

> ${UDBDIR}/${UDBBASE}—Merged_Class_Metrics . csv 


^${UDBBASE}—{c .class -filenames , cj .classmetrics , coupling}, csv \ 
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2. Converting Reports 

The following script was used to convert the project reports generated by Under¬ 


stand. Some necessary metrics could be extracted from the reports with some process¬ 
ing. 

#!/ bin /sh 

# Convert—Understand—Report—To—CSV. sh 
if [ $#-/f 1 ]; then 

echo ’’USAGE: ^$ (basename ^$0) ^ under stand —metric s — f i 1 e ” 

exit 

fi 

# From Understand , Reports —> Generate Reports , creates a file 

# with (part of it) looking like what you see below. Run this 

# converter to make the Class Metrics Report section tabular. 

# 

# Class Metrics Report 
#============================================================================= 

#Acceleration : 

^ Lines 36 

# Lines Blank 5 

# Lines Code 31 

# Lines Comment 0 

# Average Lines 7 

# Average Lines Comment 0 

# Average Complexity 1 

# Maximum Complexity 1 

# Ratio Comment/Code 0.00 

# 

function conve rt(){ 

SECTIONTOSUMMARIZE=” $ 1 ” 

INPUTFILE=”$2” 

INPUTDIR=$( dirname ”$INPUTFILE”) 

INPUTBASE=$(basename -s . txt ”$INPUTFILE”) 

THISSCRIPT=”$0” 

THISSCRIPTDIR=$ ( dirname ”$THISSCRIPT” ) 

AWK=awk 

if [ %# == 3 ]; then 

if [ ”$3” = ” ]; then 

# To Standard Out 

$AWK-f ”${THISSCRIPTDIR}/Understand-Report-to-CSV.awk” \ 
SectionToSummarize=”$SECTIONTOSUMMARIZE” ”$INPUTFILE” 

else 

# To Named File 
OUTPUTFILE=$2 

$AWK-f ”${THISSCRIPTDIR}/Understand-Report-to-CSV.awk” \ 

SectionToSummarize=”$SECTIONTOSUMMARIZE” ”$INPUTFILE” > ”$OUTPUTFILE” 
echo ” Output „$(( „$(wc„-l „$OUTPUTFILE„ I „awk„’{ print „$1 } ’) 1 „)) ^records „ to „$OUTPUTFILE” 

fi 

else 

# Generate a default file name 

OUTPUTFILE=${INPUTDIR}/${INPUTBASE}-$(echo SSECTIONTOSUMMARIZE | tr ’ ’ ’_’).csv 
$AWK — f ”$( dirname $0” )/Understand—Report—to—CSV. awk” \ 

SectionToSummarize=”$SECTIONTOSUMMARIZE” ”$INPUTFILE” > ”$OUTPUTFILE” 
echo ”Output„$((„$(wc„—1 „$OUTPUTFILE„ | „awk„ ’{print„$l}’) „1 „))„records„to 

$( tput „setaf „2)$OUTPUTFILE$( tput„sgr0)” 
fi 
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} 


# Do conversions 

echo ”Converting^Understand-reports-to-individual-csv-files ... 

convert ” Class ^Metrics ^Report” ”$1” 

convert ” Class „OO^Metrics ^Report” ”$1” 

convert ’’Program^Unit^Complexity^Report” $1 

convert ” File ^Average ^Metrics ” ”$1” 

convert ’’File^Metrics” ”$1” 

convert ’’Program^Unit^Complexity” ”$1” 


3. Running Perl Scripts 

The software Understand has a Perl API for advanced analysis. Here are the files 

used. 

#! /bin/sh 
# RunAllPerl. sh 

if [ $# != 1 ]; then 

echo ’’USAGE: ^$ (basename ^$0) ^ under stand —database . udb” 

echo Creates ^a^number^of^ metrics ^files^based^on^the^Understand^Database” 

exit 

fi 

UDB=”$1” 

THISSCRIPT=”$0” 

UDBBASE=$(basename —s .udb ”$UDB”) 

UDBDIR=$ ( dirname ”$UDB”) 

THISSCRIPTDIR=$ ( dirname ”$THISSCRIPT” ) 


PROC=l 

echo ” Detecting ^number^of ^processors ... ” ‘sysctl —n hw.ncpu‘ && PROC= ‘ s y s c 11 —n hw.ncpu‘ 
/ bin / Is ${THISSCRIPTDIR }/{ cj.classmetrics,coupling,c_class_filenames,c_halstead}.pi | \ 
xargs -n 1 -P ”$PROC” ”${THISSCRIPTDIR}/RunOnePerl. sh” ”$UDB” 

exit 


# Execute all the Perl scripts in "scripts” folder as Understand Perl modules 

#for script in ${THlSSCRIPTDlR}/{ cj-clas smetric s , coupling , c-clas s-filename s , c-halstead^. pi; do 

# scriptBase=$( basename —s .pi ” Sscript ”) 

# CSYFlLE^”${UDBDlR^/${UDBBASE\-${scriptBase }. 

# echo "Processing $( tput setaf 2) $script$ (tput sgrO) to $( tput setaf 1 )$CSVF1LE$( tput sgrO)” 

# uperl "Sscript" —db ”$UDB” —ci'v "SCSVFILE” > /dev/null 
#done 

#!/bin/sh 

# RunOnePerl. sh 

if [ $# != 2 ]; then 

echo ’’USAGE: ^$ (basename ^$0) ^ under stand —database . udb ^understand—perl — s c ri p t . pi ” 
echo ” ^^Runs ^the^Perl^script^for ^Understand ^and ^adds ^a^—csv ^command ^ 1 i ne ^argument . ” 
echo ’’^^Aids^in^using^xargs^to^speed^up^processing .” 

exit 

fi 

THISSCRIPT=”$0” 

UDB=”$1” 
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PERLSCRIPT=”$2” 

UDBBASE=$ ( basename —s . udb ”$UDB”) 

UDBDIR=$ ( dirname ”$UDB”) 

PERLSCRIPTDIR=$ ( dirname ”$PERLSCRIPT” ) 

s crip tB ase =$( basename —s .pi ”$PERLSCRIPT”) 

CSVFILE=” $ {UDBDIR } / $ {UDBBASE}-$ {scriptBasej.csv” 

echo ”Processing„$(tput„setaf„ 1) $PERLSCRIPT$ (tput„sgrO)„to„$(tput„ s e t af „ 2) $CSVEILE$ (tput„sgrO)” 
uperl ’’SPERLSCRIPT” -db ”$UDB” -csv ”$CSVFILE” > /dev/null 


4. Merge CSV 


The following script was used to merge multiple CSV files. 

#!/ bin/sh 
# MergeCsv. sh 

if [ %# -It I ]; then 

echo ’’USAGE: „$( basename _$0) „filel .csv_file2 .csv„...” 

echo ’’Merges „ two „ or „more„CSV„ files„with„the„ first ^columns „ being „common„keys ” 

exit 


function Merge () { 

CSV1=$1 

CSV2=$2 

awk-F, -V Filel=$CSVl -V File2=$CSV2 ’ 

BEGIN { 

# Read in the second file and record the header 

# in an array that we can pull out later. We w 

# referencing the first columns as the key to i 
NeedHeadersl = 1; 

NeedHeaders2 = 1; 

F2KeysCount = 0; 

while ( (getline < File2) > 0 ) { 
if( NeedHeaders2 ){ 

for( i = 2; i <= NF; i++ ){ 

Headers2 [ i ] = $i ; 

} 

Headers2 [’’count ” ] = NF — 1; 

NeedHeaders2 = 0; 

} else { 

f2data [$1 ,0] = $0; 
for( i = 2; i <= NF; i++ ){ 
f2data [$1 , i ] = $i ; 

} 

F2Keys[$l] = $1; 


} 

} # end while 
OFS=” 

} 

{ 

# Headers 

if( NeedHeadersl ){ 
printf( ”%s” , $0 ); 


■ and the data 
ill be cross — 
mite the rows. 


# Reads each row of the second file 

# Looking for the first row here 

# Loop through column titles 

# Save the column titles 

# Save how many data columns we have 

# Mark that we are done with headers 

# Save the whole row 

# Loop through the columns of data 

# Save the data in a 2D array 

# Keep separate list of keys (first column) 

# delete keys from this as they are used 

# to see if any keys appeared in the second 

# file but not in the first. 


# Looking for first row here 

# Print the whole CSV line from file 1 
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Headersl ["count”] = NF — 1; 


for( i = 2; i <= Headers2 [" count ”] +1; i++ ){ 
printf( ”,%s” , Headers2 [ i ] ); 

} 

printf( ”\n” ); 

NeedHeadersl = 0; 

} else { 

# Data 

printf ( ”%s” , $0 ); 

for( i = 2; i <= Headers2 [” count ”] +1; i++ ){ 
printf( ”,%s" , f2data[$l,i] ); 
delete F2Keys [$1 ]; 

} 

printf ( ”\n” ); 

} 

} 

END { 

# Append keys that were only in file 2 
for( key in F2Keys ){ 
printf ( ”%s” , key ); 

for( i = 2; i <= Headers 1 ["count”] +1; i++ ){ 

printf ( ” ," ); 

} 

for( i = 2; i <= Headers2 ["count”] +1; i++ ){ 
printf] ”,%s” , f2data [key , i ] ); 

delete F2Keys[key]; 

} 


# Store number of columns which we 

# may need if there were keys in the 

# second file that do not show up here. 

# Loop through columns of file 2 

# Append column titles from file 2 

# End of row 

# Done with column titles 


# Print the whole CSV line from file 1 

# Loop through columns of file 2 

# Append data from file 2 

# Remove from list of unprocessed file 2 keys 

# End of row 


# Loop through unprocessed file 2 keys 

# Write the key to the first column 

# Add blanks to all the columns for 

# filel that did not have this key. 

# Loop over columns in file 2 

# Append actual data from file 2 

# Remove from list of unprocessed keys 


} 

} 

’ $CSV1 

} 


CSVTEMP1=”$ (mktemp„—t „MergeCsv) ” 

CSVTEMP2=”$ (mktemp„—t „MergeCsv) ” 

trap ”rm„-f„’$CSVTEMPr „’$CSVTEMP2”’ 0 # EXIT 

trap "rm„-f„’$CSVTEMPr „’$CSVTEMP2’; „exit„l” 2 # INT 

trap "rm„-f„’$CSVTEMPr „’$CSVTEMP2’; „exit_l” 1 15 # HUP TERM 


# Merge a 11 CSV files 
while [ ”$1” ]; do 

echo ” Adding„$ (tput „ se t af „ 1) $1$ (tput „sgr0 ) ” >&2 
if [ -s ”$CSVTEMP1" ]; then 
#echo "Merging two CSV files" 

Merge ”$CSVTEMP1” - < ”$1” > ”$CSVTEMP2” ; 
else 

#echo "Establishing first CSV file" 
cat ”$1” > "SCSVTEMPl” 
cat ”$1” > ”$CSVTEMP2” 

fi 

cat "$CSVTEMP2” > "SCSVTEMPl” 

shift 

done 

cat "SCSVTEMPl” 
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B. SCRIPTS FOR CALCULATING AGILITY METRICS 


The metrics can be calculated with command line tools such as grep and awk. There 
is some risk of counting a line that is in a block comment (\*. . . *\), but single line 
comments (\\...) are excluded. 

1. Lines with Includes, LI 

# DeltaSD: 

grep —r osg merged—src—inc | grep ' ^include ’ | grep —v V/’ | wc —I 

# DMZ: 

grep —r osg foundation frameworks kernel | grep ' ^include ’ | grep —v V/’ | wc —I 


2. Classes with Includes, Cl 


# DeltaSD: 

grep —r osg merged—src—inc | grep ’ ^include ’ \ awk —F: ’{ print Sr } ’ | \ 

sed ’s/\(.*\)\..*/\l/’ I sort —u | grep —v ’ // ’ | wc — 1 

# DMZ: 

grep —r osg foundation frameworks kernel | grep ' #include ’ \ \ 

awk —F: ’{ print $1 }’ | sed ’s/\(.*\)\..*/\l/’ | sort —u | grep —v ’ll’ \ wc — 1 


3. Lines with References, LR 


# DeltaSD: 

grep —r —include =h<. cpp ’ osg [a—zA—Z] ’ merged—src—inc | grep —v ’//’ | wc —1 

# DMZ: 


grep —r —include =h<. cpp ’ osg [ a—zA—Z] :: ’ 


foundation frameworks kernel | grep —v ’//’ | wc —1 


4. Classes with References, CR 


# DeltaSD: 

grep —r —include =h<. cpp ’ osg [ a—zA—Z] ’ merged—src—inc | grep —v 'll' \ \ 

awk —F: ’{ print $1 }’ | sort —u | wc —1 
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# DMZ: 

grep —r —include =h<. cpp ’ osg [ a—zA—Z] :: ’ foundation frameworks kernel | \ 
grep —V ’/ / ’ I awk —F: ’{ print $1 }’ | sort —u | wc — 1 
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